Image fusion method and device, storage medium and terminal

ABSTRACT

Embodiments of this application disclose an image fusion method performed by a computing device. The method includes the following steps: obtaining source face image data of a current to-be-fused image and resource configuration information of a current to-be-fused resource, performing image recognition processing on the source face image data, to obtain source face feature points corresponding to the source face image data, and generating a source face three-dimensional grid of the source face image data according to the source face feature points, performing grid fusion by using a resource face three-dimensional grid and the source face three-dimensional grid to generate a target face three-dimensional grid, and performing face complexion fusion by using source complexion data of the source face image data and resource complexion data of resource face image data on the target face three-dimensional grid, to generate fused target face image data.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation application of PCT Application No.PCT/CN2018/116832, entitled “IMAGE FUSION METHOD AND DEVICE, STORAGEMEDIUM, AND TERMINAL” filed on Nov. 22, 2018, which claims priority toChinese Patent Application No. 201711173149.X, entitled “IMAGE FUSIONMETHOD AND DEVICE, STORAGE MEDIUM, AND TERMINAL” filed with the ChineseNational Intellectual Property Administration on Nov. 22, 2017, all ofwhich are incorporated by reference in their entirety.

FIELD OF THE TECHNOLOGY

This application relates to the field of computer technologies, and inparticular, to an image fusion method and device, a storage medium, anda terminal.

BACKGROUND OF THE DISCLOSURE

With the rapid development of computer technologies, terminal devicessuch as a smartphone, a palmtop computer, and a tablet computer may beinstalled with terminal applications for processing pictures, forexample, a camera, photoshopping software, and a social APP. Based onthe foregoing terminal applications, users may perform processing suchas adding special effects, decoration and beautification, hairdressingand makeup, and figure modelling change on original pictures (forexample, figures, landscapes, or buildings) or videos. Generally, out ofsearching for beauty or fun, people select to beautify or modify theirown face photos appropriately when posting their own photos on a socialwebsite or a webcast website.

SUMMARY

Embodiments of this application provide an image fusion method anddevice, a storage medium, and a terminal, so that by analyzing a processof performing image fusion on image data based on a three-dimensionalmodel to generate target face image data, authenticity of the finallyobtained target face image data can be improved.

A first aspect of the embodiments of this application provides an imagefusion method, performed at a computing device having a processor andmemory and a plurality of programs stored in the memory, the methodcomprising:

obtaining source face image data of a current to-be-fused image andresource configuration information of a current to-be-fused resource,the resource configuration information comprising resource face imagedata, resource complexion data, and a resource face three-dimensionalgrid;

-   -   obtaining source face feature points from the source face image        data through image recognition;    -   generating a source face three-dimensional grid of the source        face image data according to the source face feature points;    -   performing grid fusion to the resource face three-dimensional        grid and the source face three-dimensional grid to generate a        target face three-dimensional grid; and    -   performing face complexion fusion on the target face        three-dimensional grid by using source complexion data of the        source face image data and the resource complexion data of the        resource face image data, to generate fused target face image        data.

A second aspect of the embodiments of this application provides acomputing device, comprising a processor and memory, the memory storinga plurality of computer programs, wherein the computer programs, whenexecuted by the processor, perform the aforementioned image fusionmethod.

A third aspect of the embodiments of this application provides anon-transitory computer storage medium, the computer storage mediumstoring a plurality of instructions, the instructions being configuredfor, when executed by a processor a computing device, perform theaforementioned image fusion method.

BRIEF DESCRIPTION OF THE DRAWINGS

To describe the technical solutions in the embodiments of thisapplication or in the related art more clearly, the following brieflyintroduces the accompanying drawings for describing the embodiments orthe related art. Apparently, the accompanying drawings in the followingdescription show merely some embodiments of this application, and aperson of ordinary skill in the art may still derive other drawings fromthe accompanying drawings without creative efforts.

FIG. 1A is a schematic diagram of an application scenario to which theimage fusion device according to the embodiments of this application isapplicable.

FIG. 1B to FIG. 1H are schematic application diagrams of an image fusionmethod according to an embodiment of this application.

FIG. 2 is a schematic flowchart of an image fusion method according toan embodiment of this application.

FIG. 3 is a schematic diagram of positions of face reference pointsaccording to an embodiment of this application.

FIG. 4 is a schematic diagram of a face three-dimensional grid modelaccording to an embodiment of this application.

FIG. 5A and FIG. 5B are schematic diagrams of face types according to anembodiment of this application.

FIG. 6 is a schematic flowchart of another image fusion method accordingto an embodiment of this application.

FIG. 7 is a schematic flowchart of another image fusion method accordingto an embodiment of this application.

FIG. 8 is a schematic flowchart of another image fusion method accordingto an embodiment of this application.

FIG. 9 is a schematic system diagram of an image fusion method accordingto an embodiment of this application.

FIG. 10 is a schematic structural diagram of an image fusion deviceaccording to an embodiment of this application.

FIG. 11 is a schematic structural diagram of another image fusion deviceaccording to an embodiment of this application.

FIG. 12 is a schematic structural diagram of a source grid generatingmodule according to an embodiment of this application.

FIG. 13 is a schematic structural diagram of a target data generatingmodule according to an embodiment of this application.

FIG. 14 is a schematic structural diagram of a target data generatingunit according to an embodiment of this application.

FIG. 15 is a schematic structural diagram of a terminal according to anembodiment of this application.

DESCRIPTION OF EMBODIMENTS

The following clearly and completely describes the technical solutionsin the embodiments of this application with reference to theaccompanying drawings in the embodiments of this application.Apparently, the described embodiments are some of the embodiments ofthis application rather than all of the embodiments. All otherembodiments obtained by a person of ordinary skill in the art based onthe embodiments of this application without creative efforts shall fallwithin the protection scope of this application.

Generally, a process of a photo retouching terminal applicationprocessing face image data is generating, according to 2D faceinformation of a user and 2D face information of a resource, a resultimage like both the user and the face in the resource by using a certainfusion algorithm. However, when fusion is performed on a user image anda resource image based on a 2D model to generate a target result image,because image information that can be extracted by the fused user imageand resource image used as plane images reflects a poor effect of thereal face, the final fusion effect is poor, which affects authenticityof the finally obtained target result image.

If an image fusion method provided in the embodiments of thisapplication is applied to an image fusion device, for example, a photoretouching terminal application in the image fusion device, a morerealistic picture may be obtained during photo retouching. For example,the image fusion device obtains source face image data of a currentto-be-fused image and resource configuration information of a currentto-be-fused resource, where the resource configuration informationincludes resource face image data, resource complexion data, and aresource face three-dimensional grid, then performs image recognitionprocessing on the source face image data, to obtain source face featurepoints corresponding to the source face image data, and generates asource face three-dimensional grid of the source face image dataaccording to the source face feature points, then performs grid fusionby using the resource face three-dimensional grid and the source facethree-dimensional grid to generate a target face three-dimensional grid,and finally performs face complexion fusion on the target facethree-dimensional grid by using source complexion data of the sourceface image data and resource complexion data of the resource face imagedata, to generate fused target face image data. By analyzing the processof fusing the resource face three-dimensional grid and the source facethree-dimensional grid into the target face three-dimensional grid basedon a three-dimensional model, and performing complexion fusion on thetarget face three-dimensional grid to generate the target face imagedata, authenticity of the finally obtained target face image data isimproved.

FIG. 1A is a schematic diagram of an application scenario to which animage fusion device according to the embodiments of this application isapplicable. As shown in FIG. 1A, the application scenario includes animage fusion device 10 and a server 20. The image fusion device 10 isinstalled with an image fusion application client. The server 20 is, forexample, an image fusion application server, and communicates with theimage fusion application client through a network. The image fusionapplication client may perform image fusion by using the image fusionmethod of the embodiments of this application. When a user is using theimage fusion application client, an interface of the image fusionapplication client may prompt the user to select a to-be-fused resource,and after the user selects the to-be-fused resource, the image fusionapplication may obtain resource face image data of the to-be-fusedresource from the image fusion device 10 locally or from the server 20.The to-be-fused resource may be a video or an image. Next, the interfaceof the image fusion application client may prompt the user to take aselfie, to obtain source face image data of the user face, or may guidethe user to select a video or an image from an album in the image fusiondevice 10, to obtain source face image data. Then, the image fusionapplication client fuses the source face image data and the resourceface image data, to form target face image data, namely, a fused faceimage or video, to be outputted and displayed to the user. The imagefusion application client may communicate with the server 20, and updatethe resource face image data of the to-be-fused resource.

The image fusion device 10 may be further installed with an independentstand-alone image fusion application. In the following, the image fusionapplication includes the image fusion application client and thestand-alone image fusion application.

The image fusion device 10 in the embodiments of this application may beother terminal devices with an image processing function, such as atablet computer, a smartphone, a palmtop computer, and a mobile Internetdevice (MID), or may be an application server with a computer processingcapability that is used when various photo retouching terminalapplications are performing face fusion processing.

FIG. 1B to FIG. 1H are schematic application diagrams of an image fusionmethod according to an embodiment of this application. After a userenters an image fusion application on an image fusion device, aninterface of the image fusion application has an option for the user toselect a to-be-fused picture or video. Through the option, the user mayselect to take a selfie or obtain a to-be-fused picture or video fromthe image fusion device locally. The interface of the image fusionapplication may display a preview interface of the picture or videoselected by the user, as shown in FIG. 1B. After the user selects adetermining option in the preview interface, the image fusionapplication determines the picture or video selected by the user, toobtain source face image data. Next, the interface of the image fusionapplication has resource thumbnails for the user to select, as shown inFIG. 1C. The user may tap the resource thumbnails, to generate fusionresults between the user face and resource images in real time. Forexample, a resource thumbnail 30 corresponds to a resource image 31.After the user selects the resource thumbnail 30, a result image 32obtained after the user face and the resource image 31 are fused isgenerated. Next, the user may select a button 33 on the interface tosave the fusion result image. In the process, the to-be-fused face imageof the user is a front face, and the face in to-be-fused resource imageis a side face. Therefore, during the fusion, after obtaining thepicture (as shown in FIG. 1D) selected by the user, the image fusionapplication obtains a 3D image of the user face, as shown in FIG. 1E.Next, the image fusion application rotates the 3D image of the user faceto a 3D image having a face angle consistent with that in the resourceimage, as shown in FIG. 1F. The image fusion application sticks thetexture of the face image of the user onto the rotated 3D image, asshown in FIG. 1G. Finally, the fusion result image shown in FIG. 1H isgenerated.

The following describes the image fusion method provided in thisembodiment of this application in detail with reference to FIG. 2 toFIG. 9.

FIG. 2 is a schematic flowchart of an image fusion method according toan embodiment of this application. As shown in FIG. 2, the method ofthis embodiment of this application may include the following step S101to step S104.

S101. Obtain source face image data of a current to-be-fused image andresource configuration information of a current to-be-fused resource.

According to this embodiment of this application, an image fusion devicemay obtain source face image data of a current to-be-fused image andresource configuration information of a current to-be-fused resource.The source face image data may be face image data of a photo or a videotaken by a user currently by using the image fusion device or selectedfrom an album of the image fusion device. The current to-be-fusedresource may be a resource model used for photo retouching that is in aphoto retouching terminal application (for example, XXtupic, XXshop, orXXcamera) and that is selected by the user currently, for example, ananime character image or a celebrity photo. The resource configurationinformation may include 3D avatar information (for example, may be afile in obj format, and the file may include information representingresource face-related data, such as resource face image data, resourcecomplexion data, and a resource face three-dimensional grid) of thecurrent to-be-fused resource, information indicating a head effect in afinal result image (the information may include an orientation (Eulerangle pitch, yaw, and roll), a central position (a specified position ofthe final result image), scale information and camera informationmatching the scale information (for example, the information is depictedby using a perspective matrix), and the like of a resource 3D avatar ina world coordinate system), a 2D sticker and a 3D sticker, and a fusiondegree alpha of a user face and a resource face (all frames may have thesame fusion degree or each frame has a different fusion degree). Theavatar may be an image of an entire head, or may be an image of a facialregion only.

According to this embodiment of this application, the scale of theresource face image data for generating the target face image data andthe scale of the source face image data need to correspond to the samedimension, but facial concavity and plumpness of them may not becompletely consistent. Therefore, after the source face image data andthe resource face image data are obtained, the scale of the source faceimage data may be adjusted according to the scale of the resource faceimage data, to make the source face image data and the resource faceimage data correspond to the same dimension.

S102. Perform image recognition processing on the source face imagedata, to obtain source face feature points corresponding to the sourceface image data, and generate a source face three-dimensional grid ofthe source face image data according to the source face feature points.

According to this embodiment of this application, the image fusiondevice may perform image recognition processing on the source face imagedata, to obtain source face feature points corresponding to the sourceface image data. The image recognition processing may be a process ofperforming recognition and facial feature location on a user face in aphoto by using a face detection technology (for example, face detectionprovided by the cross-platform computer vision library OpenCV, the newvision service platform Face++, or the Utu face detection), the sourceface feature points may be data points that can represent facialfeatures (for example, a facial contour, an eye contour, a nose, andlips) of the source face image data.

According to this embodiment of this application, the image fusiondevice may perform image recognition processing on the source face imagedata, to obtain reference feature points of the source face image data(for example, may perform recognition and facial feature location on auser face in a photo, to obtain a certain quantity of reference featurepoints), and then perform three-dimensional depth information extractionon the reference feature points, to obtain source face feature pointscorresponding to the reference feature points. The three-dimensionaldepth information extraction may be a process of deducing feature pointsthat can reflect the source face image data in a three-dimensional modelbased on the foregoing reference feature points by matching facialfeature points of a standard three-dimensional model. The referencefeature points may be reference points indicating facial features, forexample, points such as a facial contour, an eye contour, a nose, andlips, and may be 83 reference points, or may be 68 reference pointsshown in FIG. 3. The specific point quantity may be determined bydevelopers according to requirements. The source face feature points maybe feature points that can correspond to the three-dimensional model ofthe source face image data after further deepening based on thereference feature points. For example, through the three-dimensionaldepth information extraction on the foregoing 68 or 83 reference points,1000 deepened source face feature points may be obtained, which may bevertexes of triangular patches shown in FIG. 4.

Further, the image fusion device may generate a source facethree-dimensional grid of the source face image data according to thesource face feature points. The source face three-dimensional grid maybe a 3D face grid model corresponding to the face in the source faceimage data, for example, a 3D face grid shown in FIG. 4 or similar to a3D face grid of only half of a face in FIG. 4.

S103. Perform grid fusion by using the resource face three-dimensionalgrid and the source face three-dimensional grid to generate a targetface three-dimensional grid.

According to this embodiment of this application, because scales of thesource face image data and the resource face image data are in the samedimension, dimensions of the source face three-dimensional grid and theresource face three-dimensional grid are also in the same dimension.

Specifically, the image fusion device may perform grid fusion by usingthe resource face three-dimensional grid and the source facethree-dimensional grid to generate a target face three-dimensional grid.The resource face three-dimensional grid may be similar to the foregoingsource face three-dimensional grid, may be a 3D face grid modelcorresponding to the face in the resource face image data, and may be a3D face grid shown in FIG. 4 or a 3D face grid obtained after theorientation changes according to a Euler angle in a world coordinatesystem based on a 3D face grid shown in FIG. 4. The target facethree-dimensional grid may be a 3D face grid of the target face imagedata. According to this embodiment of this application, when theresource face three-dimensional grid is the 3D face grid shown in FIG.4, the resource face three-dimensional grid finally projected on a 2Dimage may present a front face effect, and when the resource facethree-dimensional grid is the 3D face grid obtained after theorientation changes according to a Euler angle in a world coordinatesystem based on the 3D face grid shown in FIG. 4, the resource facethree-dimensional grid finally projected on a 2D image may present aside face effect.

According to this embodiment of this application, the image fusiondevice may calculate target face feature points of a target faceaccording to the face feature points in the source facethree-dimensional grid and the resource face three-dimensional grid, andthen generate a target face three-dimensional grid according to thecalculated target face feature points. For example, a 3D source facethree-dimensional grid (namely, a 3D face grid of a user) has 1000source face feature points with depth information, which are marked tobe green, a 3D resource face three-dimensional grid has 1000 resourceface feature points with depth information, which are marked to be blue,average points of each corresponding point of the 1000 face featurepoints of the user and each corresponding point of the 1000 face featurepoints of the resource (corresponding points at the same position areaveraged, and there are a total of 1000 point pairs) are marked to bered, and the finally generated 1000 red face feature points are thetarget face feature points. The foregoing 1000 red face feature pointsmay form more than 1900 triangles, and a face three-dimensional griddepicted by corresponding more than 1900 triangular patches is thetarget face three-dimensional grid. According to this embodiment of thisapplication, the image fusion device may use algorithms such as an imagedeformation using moving least squares (MLS) method, affinetransformation, and image distortion, to make facial feature positionsof the source face image data and the resource face image data tend tofacial feature positions indicated by the foregoing red face featurepoints, namely, the target face feature points, to achieve the objectiveof face fusion.

S104. Perform face complexion fusion on the target facethree-dimensional grid by using source complexion data of the sourceimage data and resource complexion data of the resource face image data,to generate fused target face image data.

According to this embodiment of this application, after generating thetarget face three-dimensional grid, the image fusion device needs toperform complexion filling on different triangular patches in the targetface three-dimensional grid to obtain the final target face image data.

Specifically, the image fusion device may perform face complexion fusionon the target face three-dimensional grid by using source complexiondata of the source face image data and resource complexion data of theresource face image data, to generate fused target face image data.According to this embodiment of this application, the source complexiondata may be a set of source pixel points forming the source face imagedata, and the resource complexion data may be a set of resource pixelpoints forming the resource face image data.

According to this embodiment of this application, if types of the sourceface three-dimensional grid and the resource face three-dimensional gridare inconsistent, grid supplementing needs to be performed on the sourceface three-dimensional grid, to generate a candidate facethree-dimensional grid. The types may be grid elements in the sourceface three-dimensional grid and the resource face three-dimensionalgrid. Candidate complexion data of the candidate face three-dimensionalgrid may be symmetrical source face image data, namely, the candidatecomplexion data of the candidate face three-dimensional grid may beregarded as the source complexion data. The image fusion device maycalculate target pixel points of the target face image data according tothe foregoing source pixel points, the resource pixel points, and afusion degree (may be a fusion value set according to an empiricalvalue, and the general value ranges from 0 to 1) in the resourceconfiguration information, to further fill the target facethree-dimensional grid according to the target pixel points to generatethe target face image data. For example, it is set that: pixels of afeature point on a triangular patch in a source face three-dimensionalgrid are: UserB, UserG, and UserR, pixels of a facial feature point at acorresponding position on the corresponding triangular patch in aresource face three-dimensional grid are: ResourceB, ResourceG, andResourceR, pixels of a feature point at the corresponding position onthe corresponding triangular patch in a target face three-dimensionalgrid are: TargetB, TargetG, and TargetR, and the fusion degree in theresource configuration information is: alpha, there are:TargetB=(1.0−alpha)*UserB+alpha*ResourceBTargetG=(1.0−alpha)*UserG+alpha*ResourceGTargetR=(1.0−alpha)*UserR+alpha*ResourceR

Therefore, each pixel value of the target face image data may beobtained, to obtain the target face image data.

According to this embodiment of this application, the finally generatedtarget face image data may be three-dimensional face image data, or maybe two-dimensional face image data. When the target face image data istwo-dimensional face image data, because the process of generating thetarget image data is implemented based on the three-dimensional model,the effect of the finally formed target face image data is morerealistic in consideration of problems such as real light and shadows.

According to this embodiment of this application, the candidatecomplexion data of the candidate face three-dimensional grid may includetwo parts, namely, complexion data of a part of the candidate facethree-dimensional grid matching the source face three-dimensional gridis the source complexion data, and complexion data of a part of thecandidate face three-dimensional grid not matching the source facethree-dimensional grid is average complexion data. The averagecomplexion data may be complexion data obtained after the sourcecomplexion data is subjected to complexion balance processing. Accordingto this embodiment of this application, the complexion balanceprocessing may be a process of removing effects such as shadows causedby light or the like from the source face image data to obtain acomplexion average value. The image fusion device may calculate thetarget pixel points of the target face image data according to candidatepixel points of the foregoing candidate complexion data, the resourcepixel points, and the fusion degree in the resource configurationinformation, to further fill the target face three-dimensional gridaccording to the target pixel points to generate the target face imagedata. The specific calculation process is consistent with the foregoingcalculation process, and details are not provided herein again.

In this embodiment of this application, source face image data of acurrent to-be-fused image and resource configuration information of acurrent to-be-fused resource are obtained, where the resourceconfiguration information includes resource face image data, resourcecomplexion data, and a resource face three-dimensional grid, then imagerecognition processing is performed on the source face image data, toobtain source face feature points corresponding to the source face imagedata, and a source face three-dimensional grid of the source face imagedata is generated according to the source face feature points, then gridfusion is performed by using the resource face three-dimensional gridand the source face three-dimensional grid to generate a target facethree-dimensional grid, and finally face complexion fusion is performedon the target face three-dimensional grid by using source complexiondata of the source face image data and resource complexion data of theresource face image data, to generate fused target face image data. Byanalyzing the process of fusing the resource face three-dimensional gridand the source face three-dimensional grid into the target facethree-dimensional grid based on a three-dimensional model, andperforming complexion fusion on the target face three-dimensional gridto generate the target face image data, authenticity of the finallyobtained target face image data is improved.

FIG. 6 is a schematic flowchart of another image fusion method accordingto an embodiment of this application. As shown in FIG. 6, the method ofthis embodiment of this application may include the following step S201to step S210.

S201. Obtain source face image data of a current to-be-fused image andresource configuration information of a current to-be-fused resource.

According to this embodiment of this application, an image fusion devicemay obtain source face image data of a current to-be-fused image andresource configuration information of a current to-be-fused resource.The source face image data may be face image data of a photo or a videotaken by a user currently by using the image fusion device or selectedfrom an album of the image fusion device. The current to-be-fusedresource may be a resource model used for photo retouching that is in aphoto retouching terminal application (for example, XXtupic, XXshop, orXXcamera) and that is selected by the user currently, for example, ananime character image or a celebrity photo. The resource configurationinformation may include 3D avatar information (for example, may be afile in obj format, and the file may include information representingresource face-related data, such as resource face image data, resourcecomplexion data, and a resource face three-dimensional grid) of thecurrent to-be-fused resource, information indicating a head effect in afinal result image (the information may include an orientation (Eulerangle pitch, yaw, and roll), a central position (a specified position ofthe final result image), scale information and camera informationmatching the scale information (for example, the information is depictedby using a perspective matrix), and the like of a resource 3D avatar ina world coordinate system), a 2D sticker and a 3D sticker, and a fusiondegree alpha of a user face and a resource face (all frames may have thesame fusion degree or each frame has a different fusion degree).

According to this embodiment of this application, the scale of theresource face image data for generating the target face image data andthe scale of the source face image data need to correspond to the samedimension, but facial concavity and plumpness of them may not becompletely consistent. Therefore, after the source face image data andthe resource face image data are obtained, the scale of the source faceimage data may be adjusted according to the scale of the resource faceimage data, to make the source face image data and the resource faceimage data correspond to the same dimension.

S202. Perform image recognition processing on the source face imagedata, to obtain reference feature points corresponding to the sourceface image data.

According to this embodiment of this application, the image fusiondevice may perform image recognition processing on the source face imagedata, to obtain reference feature points corresponding to the sourceface image data. The image recognition processing may be a process ofperforming recognition and facial feature location on a user face in aphoto by using a face detection technology (for example, face detectionprovided by the cross-platform computer vision library OpenCV, the newvision service platform Face++, or the Utu face detection). Thereference feature points may be reference points indicating facialfeatures, for example, points such as a facial contour, an eye contour,a nose, and lips, and may be 83 reference points, or may be 68 referencepoints shown in FIG. 3. The specific point quantity may be determined bydevelopers according to requirements.

S203. Perform three-dimensional depth information extraction on thereference feature points, to obtain source face feature pointscorresponding to the reference feature points, and generate a sourceface three-dimensional grid according to the source face feature points.

According to this embodiment of this application, the image fusiondevice may perform three-dimensional depth information extraction on thereference feature points, to obtain source face feature pointscorresponding to the reference feature points, and generate a sourceface three-dimensional grid according to the source face feature points.The three-dimensional depth information extraction may be a process ofdeducing feature points that can reflect the source face image data in athree-dimensional model based on the foregoing reference feature pointsby matching facial feature points of a standard three-dimensional model.The source face feature points may be further deepened points based onthe reference feature points. For example, through the three-dimensionaldepth information extraction on the foregoing 68 or 83 reference points,1000 deepened source face feature points may be obtained, which may bevertexes of triangular patches in FIG. 4. The source facethree-dimensional grid may be a 3D face grid model corresponding to theface in the source face image data, and may be a 3D grid model of a userface formed by connecting the source face feature points, for example, a3D face grid shown in FIG. 4 or similar to a 3D face grid of only halfof a face in FIG. 4.

S204. Perform grid fusion by using the source face three-dimensionalgrid and the resource face three-dimensional grid to generate a targetface three-dimensional grid.

According to this embodiment of this application, because scales of thesource face image data and the resource face image data are in the samedimension, dimensions of the source face three-dimensional grid and theresource face three-dimensional grid are also in the same dimension.

Specifically, the image fusion device may perform grid fusion by usingthe resource face three-dimensional grid and the source facethree-dimensional grid to generate a target face three-dimensional grid.According to this embodiment of this application, the resource facethree-dimensional grid may be similar to the foregoing source facethree-dimensional grid, may be a 3D face grid model corresponding to theface in the resource face image data, and may be a 3D face grid shown inFIG. 4 or a 3D face grid obtained after the orientation changesaccording to a Euler angle in a world coordinate system based on a 3Dface grid shown in FIG. 4. The target face three-dimensional grid may bea 3D face grid of the target face image data. According to thisembodiment of this application, when the resource face three-dimensionalgrid is the 3D face grid shown in FIG. 4, the resource facethree-dimensional grid finally projected on a 2D image may present afront face effect, and when the resource face three-dimensional grid isthe 3D face grid obtained after the orientation changes according to aEuler angle in a world coordinate system based on the 3D face grid shownin FIG. 4, the resource face three-dimensional grid finally projected ona 2D image may present a side face effect.

In this embodiment of this application, the image fusion device maycalculate target face feature points of a target face according to theface feature points in the source face three-dimensional grid and theresource face three-dimensional grid, and then generate a target facethree-dimensional grid according to the calculated target face featurepoints. For example, a 3D source face three-dimensional grid (namely, a3D face grid of a user) has 1000 source face feature points with depthinformation, which are marked to be green, a 3D resource facethree-dimensional grid has 1000 resource face feature points with depthinformation, which are marked to be blue, average points of eachcorresponding point of the 1000 face feature points of the user and eachcorresponding point of the 1000 face feature points of the resource(corresponding points at the same position are averaged, and there are atotal of 1000 point pairs) are marked to be red, and the finallygenerated 1000 red face feature points are the target face featurepoints. The foregoing 1000 red face feature points may form more than1900 triangles, and a face three-dimensional grid depicted bycorresponding more than 1900 triangular patches is the target facethree-dimensional grid. According to this embodiment of thisapplication, the image fusion device may use algorithms such as an MLSmethod, affine transformation, and image distortion, to make facialfeature positions of the source face image data and the resource faceimage data tend to facial feature positions indicated by the foregoingred face feature points, namely, the target face feature points, toachieve the objective of face fusion.

S205. Perform complexion balance processing on the source face imagedata, to obtain average complexion data of the source face image data.

According to this embodiment of this application, if types of the sourceface three-dimensional grid and the resource face three-dimensional gridare inconsistent, grid supplementing needs to be performed on the sourceface three-dimensional grid, to generate a candidate facethree-dimensional grid. The types may be grid elements in the sourceface three-dimensional grid and the resource face three-dimensionalgrid. Candidate complexion data of the candidate face three-dimensionalgrid may be symmetrical source face image data, namely, the candidatecomplexion data of the candidate face three-dimensional grid may beregarded as the source complexion data. For example, if the finallygenerated candidate face image data is the face image in FIG. 5A, theface complexion in FIG. 5A is consistent with the face complexion inFIG. 5B.

In this embodiment of this application, the candidate complexion data ofthe candidate face three-dimensional grid may include two parts, namely,complexion data of a part of the candidate face three-dimensional gridmatching the source face three-dimensional grid is the source complexiondata, and complexion data of a part of the candidate facethree-dimensional grid not matching the source face three-dimensionalgrid is average complexion data. Specifically, the image fusion devicemay perform complexion balance processing on the source face image data,to obtain average complexion data of the source face image data. Thecomplexion balance processing may be a process of removing effects suchas shadows caused by light or the like from the source face image datato obtain a complexion average value. The average complexion data may bea pixel point set formed by average values of pixel point data obtainedafter shadows of the source complexion data are removed.

S206. Perform complexion filling on the candidate face three-dimensionalgrid based on the source complexion data of the source face image dataand the average complexion data, to generate candidate face image data.

Specifically, the image fusion device may perform complexion filling onthe candidate face three-dimensional grid based on the source complexiondata of the source face image data and the average complexion data, togenerate candidate face image data. Complexion data of a part in thecandidate face three-dimensional grid matching the source facethree-dimensional grid may be filled by the source complexion data ofthe source face image data, and complexion data of a part in thecandidate face three-dimensional grid not matching the source facethree-dimensional grid may be filled by the average complexion data. Thecandidate complexion data of the candidate face image data may includethe source complexion data and the average complexion data. For example,if the finally generated candidate face image data is the face image inFIG. 5A, the complexion of the right face in FIG. 5A is the facecomplexion in FIG. 5B, and the complexion of the left face in FIG. 5A iscomplexion obtained after the face complexion in FIG. 5B is averaged.

S207. Perform face complexion fusion on the target facethree-dimensional grid by using the candidate complexion data of thecandidate face image data and the resource complexion data of theresource face image data, to generate fused target face image data.

According to this embodiment of this application, after generating thetarget face three-dimensional grid, the image fusion device needs toperform complexion filling on different triangular patches in the targetface three-dimensional grid to obtain the final target face image data.

Specifically, the image fusion device may perform face complexion fusionon the target face three-dimensional grid by using the candidatecomplexion data of the candidate face image data and the resourcecomplexion data of the resource face image data, to generate the fusedtarget face image data. According to this embodiment of thisapplication, the candidate complexion data may be a set of candidatepixel points forming the candidate face image data, and the resourcecomplexion data may be a set of resource pixel points forming theresource face image data.

In this embodiment of this application, the image fusion device maycalculate target pixel points based on the candidate pixel points andthe resource pixel points and by using a fusion degree, to generatetarget face image data according to the target pixel points. The fusiondegree may be a fusion value set according to an empirical value, andthe general value ranges from 0 to 1.

In a specific implementation of this embodiment of this application, itmay be set that: pixels of a feature point of a triangular patch in acandidate face three-dimensional grid are: CandidateB, CandidateG, andCandidateR, pixels of a facial feature point at a corresponding positionon the corresponding triangular patch in a resource facethree-dimensional grid are: ResourceB, ResourceG, and ResourceR, pixelsof a feature point at the corresponding position on the correspondingtriangular patch in a target face three-dimensional grid are: TargetB,TargetG, and TargetR, and the fusion degree in the resourceconfiguration information is: alpha, there are:TargetB=(1.0−alpha)*CandidateB+alpha*ResourceBTargetG=(1.0−alpha)*CandidateG+alpha*ResourceGTargetR=(1.0−alpha)*CandidateR+alpha*ResourceR

Therefore, each pixel value of the target face image data may beobtained, to obtain the target face image data.

In this embodiment of this application, by analyzing the function of theaverage complexion data in complexion fusion of the target face imagedata, authenticity of the finally obtained target face image data isincreased.

S208. Obtain a light source type corresponding to the source face imagedata according to the source complexion data of the source face imagedata, and perform effect adding processing on the target face image databy using a lighting effect corresponding to the light source type.

According to this embodiment of this application, after generating thetarget face image data, the image fusion device may obtain the lightsource type corresponding to the source face image data according to thesource complexion data of the source face image data. The image fusiondevice may compare the average complexion with a complexion of eachregion in the source face image data, to obtain a region of brightnessand the average complexion and a region of darkness and the averagecomplexion, to further deduce the light source type, for example, aplurality of point light sources or area light sources. The image fusiondevice may also collect result images of the source face image data indifferent specified lighting situations through depth learning, and thenuse the result images and the corresponding lighting situations astraining data of a deep neural network (DNN), to train a DNN model thatcan output a light source type and a light source position when apicture is given.

Further, the image fusion device may perform effect adding processing onthe target face image data by using the lighting effect corresponding tothe light source type. For example, if the light source typecorresponding to the source face image data is a point light source fromthe left face direction, the image fusion device may add a lightingeffect of the point light source in the left face direction to thefinally obtained target face image data.

In this embodiment of this application, the image fusion device mayfurther stick 2D and 3D stickers in the foregoing resource configurationinformation onto the target face image data according to thecorresponding position and zOrder, for example, wear a 3D glassessticker on the face of the generated target face image data.

S209. Adjust a current display position of the target face image databased on coordinate information indicated by the resource face imagedata.

According to this embodiment of this application, after generating thetarget face image data, the image fusion device may further adjust thecurrent display position of the target face image data based on thecoordinate information indicated by the resource face image data. Forexample, the obtained target face image data is placed on the specifiedposition according to the coordinate information (including the Eulerdirection and the central point) indicated by the resource face imagedata in the foregoing resource configuration information and theresource scale.

S210. Perform, in a case that a first display region of the target faceimage data is smaller than a second display region of the source faceimage data, face edge filling processing on a part in the second displayregion except the first display region.

According to this embodiment of this application, when the first displayregion of the finally generated target face image data is smaller thanthe second display region of the source face image data, the imagefusion device may perform face edge filling processing on a part in thesecond display region except the first display region. The first displayregion may be a range on a 2D screen to which the target face image datais mapped, the second display region may be a range on a 2D screen towhich the source face image data is mapped, and that the first displayregion is smaller than the second display region may be expressed asthat the face of the target face image data is smaller than the face ofthe source face image data. The face edge filling processing may befilling the part in the second display region except the first displayregion by using a filling algorithm (for example, the image restorationalgorithm Inpainting provided by the OpenCV).

Further, after processing the additional technical effects on the targetface image data completely, the image fusion device may output anddisplay the finally obtained target face image data.

According to this embodiment of this application, after step S208 tostep S210 are performed, the finally generated target face image datamay be three-dimensional face image data, or may be two-dimensional faceimage data. When the target face image data is two-dimensional faceimage data, because the process of generating the target image data isimplemented based on the three-dimensional model, the effect of thefinally formed target face image data is more realistic in considerationof problems such as real light and shadows.

When step S208 to step S210 are performed, one or more steps thereof maybe selected to be performed simultaneously.

In this embodiment of this application, by adding real lighting effectsto the generated target face image data, and adjusting a displayposition of face image data and filling a face edge region, real effectsof the finally outputted target face image data are further increased.

In a specific implementation of this embodiment of this application, theperforming face complexion fusion on the target face three-dimensionalgrid by using the candidate complexion data of the candidate face imagedata and the resource complexion data of the resource face image data,to generate fused target face image data may include the followingseveral steps, as shown in FIG. 7:

S301. Obtain candidate pixel points in the candidate complexion data andresource pixel points in the resource complexion data.

According to this embodiment of this application, the candidatecomplexion data may be a set of candidate pixel points forming thecandidate face image data, and the resource complexion data may be a setof resource pixel points forming the resource face image data. The imagefusion device obtains candidate pixel points in the candidate complexiondata and resource pixel points in the resource complexion data.

S302. Calculate target pixel points based on the candidate pixel pointsand the resource pixel points and by using a fusion degree, to generatetarget face image data according to the target pixel points.

Specifically, for the specific process of the image fusion deviceobtaining the target face image data based on the candidate pixel pointsand the resource pixel points and by using the fusion degree, referencemay be made to the description in step S207, and details are notprovided herein again.

In this embodiment of this application, by using a complexion fusionprocess accurate to pixel points, accuracy of the complexion fusion ofthe target face image data is improved.

In this embodiment of this application, source face image data of acurrent to-be-fused image and resource configuration information of acurrent to-be-fused resource are obtained, where the resourceconfiguration information includes resource face image data, resourcecomplexion data, and a resource face three-dimensional grid, then imagerecognition processing is performed on the source face image data, toobtain source face feature points corresponding to the source face imagedata, and a source face three-dimensional grid of the source face imagedata is generated according to the source face feature points, then gridfusion is performed by using the resource face three-dimensional gridand the source face three-dimensional grid to generate a target facethree-dimensional grid, and finally face complexion fusion is performedon the target face three-dimensional grid by using source complexiondata of the source face image data and resource complexion data of theresource face image data, to generate fused target face image data. Byanalyzing the process of fusing the resource face three-dimensional gridand the source face three-dimensional grid into the target facethree-dimensional grid based on a three-dimensional model, andperforming complexion fusion on the target face three-dimensional gridto generate target face image data, authenticity of the finally obtainedtarget face image data is improved. By analyzing the function of theaverage complexion data in complexion fusion of the target face imagedata, authenticity of the finally obtained target face image data isincreased. By adding real lighting effects to the generated target faceimage data, and adjusting a display position of face image data andfilling a face edge region, real effects of the finally outputted targetface image data are further increased. By using a complexion fusionprocess accurate to pixel points, accuracy of the complexion fusion ofthe target face image data is improved.

When face fusion is performed based on a 2D model, generally thereexists a situation in which a face angle of user image data and a faceangle of resource face image data do not match completely. For example,if the user image is half of the face, the resource image is the frontface or the user head turning left, the resource head turning right, andthe like. When the foregoing situation exists, the face fusion algorithmin the related art can obtain less user face information, and duringface fusion, the less user face information will affect the finalmatching result, causing poor authenticity of the generated targetresult image.

To resolve the foregoing problem, an embodiment of this applicationprovides a schematic flowchart of another image fusion method. As shownin FIG. 8, the method of this embodiment of this application may includethe following step S401 to step S404.

S401. Obtain source face image data of a current to-be-fused image andresource configuration information of a current to-be-fused resource.

Specifically, an image fusion device may obtain source face image dataof a current to-be-fused image and resource configuration information ofa current to-be-fused resource. For the detailed obtaining process,reference may be made to the related description in step S201, anddetails are not provided herein again.

S402. Perform image recognition processing on the source face imagedata, to obtain source face feature points corresponding to the sourceface image data, and generate a source face three-dimensional grid ofthe source face image data according to the source face feature points.

Specifically, for the process of the image fusion device generating thesource face three-dimensional grid, reference may be made to the relateddescription in the foregoing step S202 to step S203, and details are notprovided herein again.

S403. Perform grid supplementing on the source face three-dimensionalgrid according to the symmetry of the source face image data in a caseof detecting that types of the source face three-dimensional grid andthe resource face three-dimensional grid are inconsistent, generate acandidate face three-dimensional grid whose type is consistent with thatof the resource face three-dimensional grid, and perform grid fusion byusing the candidate face three-dimensional grid and the resource facethree-dimensional grid to generate a target face three-dimensional grid.

According to this embodiment of this application, because scales of thesource face image data and the resource face image data are in the samedimension, dimensions of the source face three-dimensional grid and theresource face three-dimensional grid are also in the same dimension.

According to this embodiment of this application, the resource facethree-dimensional grid may be similar to the foregoing source facethree-dimensional grid, may be a 3D face grid model corresponding to theface in the resource face image data, and may be a 3D face grid shown inFIG. 4 or a 3D face grid obtained after the orientation changesaccording to a Euler angle in a world coordinate system based on a 3Dface grid shown in FIG. 4. The target face three-dimensional grid may bea 3D face grid of the target face image data. According to thisembodiment of this application, when the resource face three-dimensionalgrid is the 3D face grid shown in FIG. 4, the resource facethree-dimensional grid finally projected on a 2D image may present afront face effect, and when the resource face three-dimensional grid isthe 3D face grid obtained after the orientation changes according to aEuler angle in a world coordinate system based on the 3D face grid shownin FIG. 4, the resource face three-dimensional grid finally projected ona 2D image may present a side face effect.

According to this embodiment of this application, only when the types ofthe source face three-dimensional grid and the resource facethree-dimensional grid are consistent, a target face three-dimensionalgrid with good authenticity that is the same as a facial display regionof the source face three-dimensional grid and the resource facethree-dimensional grid can be generated according to a fusion algorithm.According to this embodiment of this application, that the types areconsistent may be that grid elements in the source facethree-dimensional grid and the resource face three-dimensional grid areconsistent, and the grid elements may be grid orientations or griddisplay regions of the source face three-dimensional grid and theresource face three-dimensional grid. For example, when the facialdisplay region indicated by the source face three-dimensional grid andthe facial display region indicated by the resource facethree-dimensional grid are consistent, and the source facethree-dimensional grid and the resource face three-dimensional grid aresimilar to the standard front face shown in FIG. 5A or similar to theside face shown in FIG. 5B, it may be regarded that the types of thesource face three-dimensional grid and the resource facethree-dimensional grid are consistent. When the types of the source facethree-dimensional grid and the resource face three-dimensional grid areinconsistent (for example, the source face three-dimensional grid issimilar to the side face shown in FIG. 5B, and the resource facethree-dimensional grid is similar to the standard front face shown inFIG. 5A), the image fusion device may first perform grid supplementingon the source face three-dimensional grid.

Specifically, when the types of the source face three-dimensional gridand the resource face three-dimensional grid are inconsistent, the imagefusion device may perform grid supplementing on the source facethree-dimensional grid according to the symmetry of the source faceimage data, to generate a candidate face three-dimensional grid whosetype is consistent with that of the resource face three-dimensionalgrid. In a normal situation, a face of a person is symmetrical (nuancesare ignored), and when the source face image data is side face imagedata (for example, the image in FIG. 5B), the source facethree-dimensional grid generated after the image fusion device extractsthe source face feature points is also for a side face. In this case,the image fusion device may supplement the source face image data as astandard front face (for example, the image in FIG. 5A is obtained afterthe image in FIG. 5B is supplemented) according to the face symmetryprinciple, and then generate a candidate face three-dimensional gridwhose type is consistent with that of the resource facethree-dimensional grid (the face three-dimensional grid corresponding toFIG. 5A is the candidate face three-dimensional grid) according to thesupplemented source face image data.

According to this embodiment of this application, left and right facialexpressions of image data of the standard front face obtained after theimage fusion device supplements the source face image data (side faceimage data) according to the face symmetry principle are consistent, andleft and right facial expressions of the further obtained candidate facethree-dimensional grid are also consistent.

In this embodiment of this application, when the resource facethree-dimensional grid is a standard front face whose left and rightfacial expressions are inconsistent, and the source face image data isside face image data, left and right facial expressions of the candidateface three-dimensional grid obtained after grid supplementing isperformed on the source face three-dimensional grid according to thesymmetry of the source face image data are inconsistent with left andright facial expressions of the resource face three-dimensional grid(for example, the left eye of the resource face indicated by theresource face image data is opened, and the right eye is closed, but theleft eye of the user face indicated by the source face image dataincluding only the left side face is opened. In this case, the right eyeof the source face three-dimensional grid supplemented according to thesymmetry of the source face image data is also opened, which isinconsistent with the right eye of the resource face three-dimensionalgrid). In this case, the image fusion device cannot perform grid fusionby using the candidate face three-dimensional grid and the foregoingresource face three-dimensional grid to generate the target facethree-dimensional grid. For the foregoing situation, the image fusiondevice may adjust the left and right facial expressions of the candidateface three-dimensional grid to be consistent with the expression of theresource face three-dimensional grid by using an expression migrationalgorithm, so that facial expressions of the finally obtained candidateface three-dimensional grid and the foregoing resource facethree-dimensional grid are consistent.

Further, the image fusion device may perform grid fusion by using thecandidate face three-dimensional grid and the resource facethree-dimensional grid, to generate the target face three-dimensionalgrid. According to this embodiment of this application, the types of thecandidate face three-dimensional grid and the resource facethree-dimensional grid are consistent, namely, the resource facethree-dimensional grid and the candidate face three-dimensional grid onthe same facial position have corresponding feature points. For example,the candidate face three-dimensional grid includes feature points ofcorners of two eyes, and the resource face three-dimensional grid alsoincludes feature points of corners of two eyes. The target facethree-dimensional grid may be a 3D face grid of the target face imagedata.

In this embodiment of this application, the image fusion device maycalculate target face feature points of a target face according to theface feature points in the candidate face three-dimensional grid and theresource face three-dimensional grid, and then generate the target facethree-dimensional grid according to the calculated target face featurepoints. For example, a 3D candidate face three-dimensional grid (namely,a 3D face grid of a user) has 1000 source face feature points with depthinformation, which are marked to be green, a 3D resource facethree-dimensional grid has 1000 resource face feature points with depthinformation, which are marked to be blue, average points of eachcorresponding point of the 1000 face feature points of the user and eachcorresponding point of the 1000 face feature points of the resource(corresponding points at the same position are averaged, and there are atotal of 1000 point pairs) are marked to be red, and the finallygenerated 1000 red face feature points are the target face featurepoints. The foregoing 1000 red face feature points may form more than1900 triangles, and a face three-dimensional grid depicted bycorresponding more than 1900 triangular patches is the target facethree-dimensional grid. According to this embodiment of thisapplication, the image fusion device may use algorithms such as an MLSmethod, affine transformation, and image distortion, to make facialfeature positions of the source face image data and the resource faceimage data tend to facial feature positions indicated by the foregoingred face feature points, namely, the target face feature points, toachieve the objective of face fusion.

S404. Perform face complexion fusion on the target facethree-dimensional grid by using source complexion data of the sourceface image data and resource complexion data of the resource face imagedata, to generate fused target face image data.

Specifically, for the process of the image fusion device generating thetarget face image data, reference may be made to the related descriptionin the foregoing step S205 to step S210, and details are not providedherein again.

In this embodiment of this application, source face image data of acurrent to-be-fused image and resource configuration information of acurrent to-be-fused resource are obtained, then image recognitionprocessing is performed on the source face image data, to obtain sourceface feature points corresponding to the source face image data, and asource face three-dimensional grid of the source face image data isgenerated according to the source face feature points. Then, when it isdetected that types of the source face three-dimensional grid and theresource face three-dimensional grid are inconsistent, gridsupplementing is performed on the source face three-dimensional gridaccording to the symmetry of the source face image data, to generate acandidate face three-dimensional grid whose type is consistent with thatof the resource face three-dimensional grid, and grid fusion isperformed by using the candidate face three-dimensional grid and theresource face three-dimensional grid to generate a target facethree-dimensional grid. Finally, face complexion fusion is performed onthe target face three-dimensional grid by using source complexion dataof the source face image data and resource complexion data of resourceface image data, to generate fused target face image data. Even if thereexists a situation in which a face angle of user image data and a faceangle of the resource face image data do not match completely, athree-dimensional avatar of the user can be created, to deduce a fullavatar of the user, and fusion at a three-dimensional level is betterperformed on the source face image data and the fusion resourcesaccording to the full avatar of the user.

The resource face image data in the foregoing method embodiment is aframe of resource data in a related terminal application, and the entireface fusion process is applicable to single-frame real-time processing,for example, may be used in a camera to preview in real time the lookobtained after the user is fused to become another resource image. Whena plurality of frames in a resource need to be processed, a plurality offrames of pictures in a video are processed circularly frame by frame toobtain a final face fused video.

The following describes a specific implementation process of a facefusion system by using an example, as shown in FIG. 9:

S500. Enter a system.

According to this embodiment of this application, the system is, forexample, a system of a photo retouching terminal application. A useropens a photo retouching terminal application on an image fusion device,and enters a home page of the photo retouching terminal application.

S501. Obtain an N^(th) frame of resource face image data and currentsource face image data.

Specifically, the home page of the photo retouching terminal applicationmay have a prompt for prompting the user to select a resource. Beforeperforming face fusion, the image fusion device may obtain resourceconfiguration information of a frame (for example, an N^(th) frame) ofthe resource in the photo retouching terminal application (for example,XXtupic, XXshop, or XXcamera) according to the resource selected by theuser. In a case that the obtained resource has a total of M frames, thevalue range of N is M≥N≥1, where N and M are positive integers. Theimage fusion device may start processing from the first frame in theresource, namely, in this case, N=1. The resource configurationinformation may include 3D avatar information (for example, may be afile in obj format, and the file may include information representingresource face-related data, such as resource face image data, resourcecomplexion data, and a resource face three-dimensional grid) of thecurrent to-be-fused resource, information indicating a head effect in afinal result image (the information may include an orientation (Eulerangle pitch, yaw, and roll), a central position (a specified position ofthe final result image), scale information and camera informationmatching the scale information (for example, the information is depictedby using a perspective matrix), and the like of a resource 3D avatar ina world coordinate system), a 2D sticker and a 3D sticker, and a fusiondegree alpha of a user face and a resource face (all frames may have thesame fusion degree or each frame has a different fusion degree).According to this embodiment of this application, the photo retouchingterminal application may guide the user to take a selfie or a video, orselect a photo or a video from an album. The process may be performedbefore or after the user selects the resource. The image fusion devicemay obtain source face image data of the photo or the video currentlytaken by the user or selected from the album.

According to this embodiment of this application, the scale of theresource face image data for generating the target face image data andthe scale of the source face image data need to correspond to the samedimension, but facial concavity and plumpness of them may not becompletely consistent. Therefore, after the source face image data andthe resource face image data are obtained, the scale of the source faceimage data may be adjusted according to the scale of the resource faceimage data, to make the source face image data and the resource faceimage data correspond to the same dimension.

S502. Generate a target face three-dimensional grid.

Specifically, after obtaining the resource face image data and thesource face image data, the image fusion device may further analyze theresource face image data and the source face image data to generate atarget face three-dimensional grid. According to this embodiment of thisapplication, the target face three-dimensional grid may be a facialthree-dimensional grid model corresponding to a face in target faceimage data generated by the final face fusion. The specificimplementation process may be implemented in steps S5021 to S5025:

S5021. Obtain source face feature points.

Specifically, the image fusion device may perform image recognitionprocessing on the source face image data, to obtain reference featurepoints of the source face image data, and then perform depth informationextraction on the reference feature points, to obtain source facefeature points corresponding to the reference feature points. For thespecific implementation process, reference may be made to the detaileddescription in step S202 and step S203, and details are not providedherein again.

S5022. Generate a source face three-dimensional grid.

Specifically, the image fusion device may connect the foregoing sourceface feature points into a source face grid formed by many triangularpatches. For the specific implementation process, reference may be madeto the detailed description in step S203, and details are not providedherein again.

S5023. Read resource configuration information.

According to this embodiment of this application, before performing facefusion, the image fusion device has obtained resource configurationinformation of the N^(th) frame of the resource. In this case, the imagefusion device may read related data in the resource configurationinformation.

S5024. Supplement the source face three-dimensional grid.

According to this embodiment of this application, when the types of thesource face three-dimensional grid and the resource facethree-dimensional grid are inconsistent (for example, the source facethree-dimensional grid is similar to the side face shown in FIG. 5B, andthe resource face three-dimensional grid is similar to the standardfront face shown in FIG. 5A), the image fusion device may first performgrid supplementing on the source face three-dimensional grid. For thespecific implementation process, reference may be made to the performinggrid supplementing according to the face symmetry in step S204, anddetails are not provided herein again.

S5025. Fuse the source face three-dimensional grid and the resource facethree-dimensional grid.

According to this embodiment of this application, the resourceconfiguration information obtained in step S5023 includes the resourceface three-dimensional grid, and the image fusion device may performgrid fusion on the source face three-dimensional grid and the resourceface three-dimensional grid to generate the target facethree-dimensional grid. For the specific fusion process, reference maybe made to the detailed process in step S204, and details are notprovided herein again.

S503. Complexion fusion on target face image data.

According to this embodiment of this application, after generating thetarget face three-dimensional grid, the image fusion device needs toperform complexion filling on different triangular patches in the targetface three-dimensional grid to obtain the final target face image data.The specific process of performing complexion fusion on the target faceimage data may be implemented in steps S5031 to S5033:

S5031. Obtain average complexion data.

Specifically, the image fusion device may perform complexion balanceprocessing on the source face image data, to obtain average complexiondata of the source face image data. For the specific process, referencemay be made to the description in step S205, and details are notprovided herein again.

S5032. Complexion fusion.

Specifically, the image fusion device may perform complexion filling onthe candidate face three-dimensional grid based on the source complexiondata of the source face image data and the average complexion data, togenerate candidate face image data, and then perform face complexionfusion on the target face three-dimensional grid by using the candidatecomplexion data of the candidate face image data and the resourcecomplexion data of the resource face image data, to generate fusedtarget face image data. For the specific implementation processes,reference may be made to the detailed description in step S206 and stepS207, and details are not provided herein again.

S504. Post processing.

According to this embodiment of this application, the post processingmay be later adjustment on the generated target face image data, so thatthe effect of the finally outputted target face image data is morerealistic. Specifically, the processing processes described in stepS5041 and step S5042 may be included.

S5041. Lighting rendering and position adjustment.

Specifically, the image fusion device may perform face complexion fusionon the target face three-dimensional grid by using candidate complexiondata of the candidate face image data and resource complexion data ofthe resource face image data, to generate fused target face image data.Meanwhile, the image fusion device may adjust a current display positionof the target face image data based on coordinate information indicatedby the resource face image data. For the specific implementationprocesses, reference may be made to the detailed description in stepS208 and step S209, and details are not provided herein again.

S5042. Face edge filling.

Specifically, when the first display region of the target face imagedata is smaller than the second display region of the source face imagedata, the image fusion device may perform face edge filling processingon a part in the second display region except the first display region.For the specific implementation process, reference may be made to thedetailed description in step S210, and details are not provided hereinagain.

S505. Result output.

According to this embodiment of this application, after processing theadditional technical effects on the target face image data completely,the image fusion device may output and display the finally obtainedtarget face image data. In a case that face fusion is performed on animage, because the entire resource has only one frame, after theforegoing steps are processed completely, the target face image data maybe saved as a picture file in a specified format, and outputted anddisplayed to the user.

S506. If the face fusion is for a video, detect whether all frames havebeen processed completely.

According to this embodiment of this application, if the face fusion isfor a video, the image fusion device detects whether all M frames in theobtained resource have been processed completely through the foregoingsteps S501 to 505. If detecting that all the M frames in the obtainedresource have been processed completely, the image fusion device maysave the target face image data as a video file in a specified format,and present a final fused video on an interface of the photo retouchingterminal application. If the M frames have not been processedcompletely, the N^(th) frame of the finally obtained target face imagedata needs to be written into a video file, and processing of theforegoing steps S501 to S505 is performed on an (N+1)^(th) frame (a nextframe of the N^(th) frame), which is followed by recycling to obtain thefinal fused video.

S507. Exit the system when all the frames have been processedcompletely.

In this embodiment of this application, source face image data of acurrent to-be-fused image and resource configuration information of acurrent to-be-fused resource are obtained, where the resourceconfiguration information includes resource face image data, resourcecomplexion data, and a resource face three-dimensional grid, then imagerecognition processing is performed on the source face image data, toobtain source face feature points corresponding to the source face imagedata, and a source face three-dimensional grid of the source face imagedata is generated according to the source face feature points, then gridfusion is performed by using the resource face three-dimensional gridand the source face three-dimensional grid to generate a target facethree-dimensional grid, and finally face complexion fusion is performedon the target face three-dimensional grid by using source complexiondata of the source face image data and resource complexion data of theresource face image data, to generate fused target face image data. Byanalyzing the process of fusing the resource face three-dimensional gridand the source face three-dimensional grid into the target facethree-dimensional grid based on a three-dimensional model, andperforming complexion fusion on the target face three-dimensional gridto generate target face image data, authenticity of the finally obtainedtarget face image data is improved. By analyzing the function of theaverage complexion data in complexion fusion of the target face imagedata, authenticity of the finally obtained target face image data isincreased. By adding real lighting effects to the generated target faceimage data, and adjusting a display position of face image data andfilling a face edge region, real effects of the finally outputted targetface image data are further increased. By using a complexion fusionprocess accurate to pixel points, accuracy of the complexion fusion ofthe target face image data is improved.

The following describes the image fusion device provided in theembodiments of this application in detail with reference to FIG. 10 toFIG. 14. The devices shown in FIG. 10 to FIG. 14 is configured toperform the methods in the embodiments shown in FIG. 2 and FIG. 9 ofthis application. For ease of description, only a part related to theembodiments of this application is shown. For specific technical detailsthat are not disclosed, refer to the embodiments shown in FIG. 2 to FIG.9 of this application.

FIG. 10 is a schematic structural diagram of an image fusion deviceaccording to an embodiment of this application. As shown in FIG. 10, theimage fusion device 1 of this embodiment of this application mayinclude: a data obtaining module 11, a source grid generating module 12,a target grid generating module 13, and a target data generating module14.

The data obtaining module 11 is configured to obtain source face imagedata of a current to-be-fused image and resource configurationinformation of a current to-be-fused resource.

In specific implementation, the image fusion device 1 may obtain sourceface image data of a current to-be-fused image and resourceconfiguration information of a current to-be-fused resource. The sourceface image data may be face image data of a photo or a video taken by auser currently by using the image fusion device 1 or selected from analbum of the image fusion device 1. The current to-be-fused resource maybe a resource model used for photo retouching that is in a photoretouching terminal application (for example, XXtupic, XXshop, orXXcamera) and that is selected by the user currently, for example, ananime character image or a celebrity photo. The resource configurationinformation may include 3D avatar information (for example, may be afile in obj format, and the file may include information representingresource face-related data, such as resource face image data, resourcecomplexion data, and a resource face three-dimensional grid) of thecurrent to-be-fused resource, information indicating a head effect in afinal result image (the information may include an orientation (Eulerangle pitch, yaw, and roll), a central position (a specified position ofthe final result image), scale information and camera informationmatching the scale information (for example, the information is depictedby using a perspective matrix), and the like of a resource 3D avatar ina world coordinate system), a 2D sticker and a 3D sticker, and a fusiondegree alpha of a user face and a resource face (all frames may have thesame fusion degree or each frame has a different fusion degree).

According to this embodiment of this application, the scale of theresource face image data for generating the target face image data andthe scale of the source face image data need to correspond to the samedimension, but facial concavity and plumpness of them may not becompletely consistent. Therefore, after the source face image data andthe resource face image data are obtained, the scale of the source faceimage data may be adjusted according to the scale of the resource faceimage data, to make the source face image data and the resource faceimage data correspond to the same dimension.

The source grid generating module 12 is configured to perform imagerecognition processing on the source face image data, to obtain sourceface feature points corresponding to the source face image data, andgenerate a source face three-dimensional grid of the source face imagedata according to the source face feature points.

According to this embodiment of this application, the source gridgenerating module 12 may perform image recognition processing on thesource face image data, to obtain source face feature pointscorresponding to the source face image data. The image recognitionprocessing may be a process of performing recognition and facial featurelocation on a user face in a photo by using a face detection technology(for example, face detection provided by the cross-platform computervision library OpenCV, the new vision service platform Face++, or theUtu face detection), the source face feature points may be data pointsthat can represent facial features (for example, a facial contour, aneye contour, a nose, and lips) of the source face image data.

According to this embodiment of this application, the source gridgenerating module 12 may perform image recognition processing on thesource face image data, to obtain reference feature points of the sourceface image data (for example, may perform recognition and facial featurelocation on a user face in a photo, to obtain a certain quantity ofreference feature points), and then perform three-dimensional depthinformation extraction on the reference feature points, to obtain sourceface feature points corresponding to the reference feature points. Thethree-dimensional depth information extraction may be a process ofdeducing feature points that can reflect the source face image data in athree-dimensional model based on the foregoing reference feature pointsby matching facial feature points of a standard three-dimensional model.The reference feature points may be reference points indicating facialfeatures, for example, points such as a facial contour, an eye contour,a nose, and lips, and may be 83 reference points, or may be 68 referencepoints shown in FIG. 3. The specific point quantity may be determined bydevelopers according to requirements. The source face feature points maybe feature points that can correspond to the three-dimensional model ofthe source face image data after further deepening based on thereference feature points. For example, through the three-dimensionaldepth information extraction on the foregoing 68 or 83 reference points,1000 deepened source face feature points may be obtained, which may bevertexes of triangular patches shown in FIG. 4.

Further, the source grid generating module 12 may generate a source facethree-dimensional grid of the source face image data according to thesource face feature points. The source face three-dimensional grid maybe a 3D face grid model corresponding to the face in the source faceimage data, for example, a 3D face grid shown in FIG. 4 or similar to a3D face grid of only half of a face in FIG. 4.

The target grid generating module 13 is configured to perform gridfusion by using the resource face three-dimensional grid and the sourceface three-dimensional grid to generate a target face three-dimensionalgrid.

According to this embodiment of this application, because scales of thesource face image data and the resource face image data are in the samedimension, dimensions of the source face three-dimensional grid and theresource face three-dimensional grid are also in the same dimension.

In specific implementation, the target grid generating module 13 mayperform grid fusion by using the resource face three-dimensional gridand the source face three-dimensional grid to generate a target facethree-dimensional grid. The resource face three-dimensional grid may besimilar to the foregoing source face three-dimensional grid, may be a 3Dface grid model corresponding to the face in the resource face imagedata, and may be a 3D face grid shown in FIG. 4 or a 3D face gridobtained after the orientation changes according to a Euler angle in aworld coordinate system based on a 3D face grid shown in FIG. 4. Thetarget face three-dimensional grid may be a 3D face grid of the targetface image data. According to this embodiment of this application, whenthe resource face three-dimensional grid is the 3D face grid shown inFIG. 4, the resource face three-dimensional grid finally projected on a2D image may present a front face effect, and when the resource facethree-dimensional grid is the 3D face grid obtained after theorientation changes according to a Euler angle in a world coordinatesystem based on the 3D face grid shown in FIG. 4, the resource facethree-dimensional grid finally projected on a 2D image may present aside face effect.

According to this embodiment of this application, the target gridgenerating module 13 may calculate target face feature points of atarget face according to the face feature points in the source facethree-dimensional grid and the resource face three-dimensional grid, andthen generate a target face three-dimensional grid according to thecalculated target face feature points. For example, a 3D source facethree-dimensional grid (namely, a 3D face grid of a user) has 1000source face feature points with depth information, which are marked tobe green, a 3D resource face three-dimensional grid has 1000 resourceface feature points with depth information, which are marked to be blue,average points of each corresponding point of the 1000 face featurepoints of the user and each corresponding point of the 1000 face featurepoints of the resource (corresponding points at the same position areaveraged, and there are a total of 1000 point pairs) are marked to bered, and the finally generated 1000 red face feature points are thetarget face feature points. The foregoing 1000 red face feature pointsmay form more than 1900 triangles, and a face three-dimensional griddepicted by corresponding more than 1900 triangular patches is thetarget face three-dimensional grid. According to this embodiment of thisapplication, the image fusion device 1 may use algorithms such as an MLSmethod, affine transformation, and image distortion, to make facialfeature positions of the source face image data and the resource faceimage data tend to facial feature positions indicated by the foregoingred face feature points, namely, the target face feature points, toachieve the objective of face fusion.

The target data generating module 14 is configured to perform facecomplexion fusion on the target face three-dimensional grid by usingsource complexion data of the source image data and resource complexiondata of the resource face image data, to generate fused target faceimage data.

According to this embodiment of this application, after generating thetarget face three-dimensional grid, the image fusion device 1 needs toperform complexion filling on different triangular patches in the targetface three-dimensional grid to obtain the final target face image data.

In specific implementation, the target data generating module 14 mayperform face complexion fusion on the target face three-dimensional gridby using source complexion data of the source image data and resourcecomplexion data of the resource face image data, to generate fusedtarget face image data. According to this embodiment of thisapplication, the source complexion data may be a set of source pixelpoints forming the source face image data, and the resource complexiondata may be a set of resource pixel points forming the resource faceimage data.

According to this embodiment of this application, if types of the sourceface three-dimensional grid and the resource face three-dimensional gridare inconsistent, grid supplementing needs to be performed on the sourceface three-dimensional grid, to generate candidate facethree-dimensional grid. The types may be grid elements in the sourceface three-dimensional grid and the resource face three-dimensionalgrid. Candidate complexion data of the candidate face three-dimensionalgrid may be symmetrical source face image data, namely, the candidatecomplexion data of the candidate face three-dimensional grid may beregarded as the source complexion data. The target data generatingmodule 14 may calculate target pixel points of the target face imagedata according to the foregoing source pixel points, the resource pixelpoints, and a fusion degree in the resource configuration information(may be a fusion degree value set according to an empirical value, andthe general value ranges from 0 to 1), to further fill the target facethree-dimensional grid according to the target pixel points to generatethe target face image data. For example, it is set that: pixels of afeature point on a triangular patch in a source face three-dimensionalgrid are: UserB, UserG, and UserR, pixels of a facial feature point at acorresponding position on the corresponding triangular patch in aresource face three-dimensional grid are: ResourceB, ResourceG, andResourceR, pixels of a feature point at the corresponding position onthe corresponding triangular patch in a target face three-dimensionalgrid are: TargetB, TargetG, and TargetR, and the fusion degree in theresource configuration information is: alpha, there are:TargetB=(1.0−alpha)*UserB+alpha*ResourceBTargetG=(1.0−alpha)*UserG+alpha*ResourceGTargetR=(1.0−alpha)*UserR+alpha*ResourceR

Therefore, each pixel value of the target face image data may beobtained, to obtain the target face image data.

According to this embodiment of this application, the finally generatedtarget face image data may be three-dimensional face image data, or maybe two-dimensional face image data. When the target face image data istwo-dimensional face image data, because the process of generating thetarget image data is implemented based on the three-dimensional model,the effect of the finally formed target face image data is morerealistic in consideration of problems such as real light and shadows.

According to this embodiment of this application, the candidatecomplexion data of the candidate face three-dimensional grid may includetwo parts, namely, complexion data of a part of the candidate facethree-dimensional grid matching the source face three-dimensional gridis the source complexion data, and complexion data of a part of thecandidate face three-dimensional grid not matching the source facethree-dimensional grid is average complexion data. The averagecomplexion data may be complexion data obtained after the sourcecomplexion data is subjected to complexion balance processing. Accordingto this embodiment of this application, the complexion balanceprocessing may be a process of removing effects such as shadows causedby light or the like from the source face image data to obtain acomplexion average value. The target data generating module 14 maycalculate the target pixel points of the target face image dataaccording to candidate pixel points of the foregoing candidatecomplexion data, the resource pixel points, and the fusion degree in theresource configuration information, to further fill the target facethree-dimensional grid according to the target pixel points to generatethe target face image data. The specific calculation process isconsistent with the foregoing calculation process, and details are notprovided herein again.

In this embodiment of this application, source face image data of acurrent to-be-fused image and resource configuration information of acurrent to-be-fused resource are obtained, where the resourceconfiguration information includes resource face image data, resourcecomplexion data, and a resource face three-dimensional grid, then imagerecognition processing is performed on the source face image data, toobtain source face feature points corresponding to the source face imagedata, and a source face three-dimensional grid of the source face imagedata is generated according to the source face feature points, then gridfusion is performed by using the resource face three-dimensional gridand the source face three-dimensional grid to generate a target facethree-dimensional grid, and finally face complexion fusion is performedon the target face three-dimensional grid by using source complexiondata of the source face image data and resource complexion data of theresource face image data, to generate fused target face image data. Byanalyzing the process of fusing the resource face three-dimensional gridand the source face three-dimensional grid into the target facethree-dimensional grid based on a three-dimensional model, andperforming complexion fusion on the target face three-dimensional gridto generate target face image data, authenticity of the finally obtainedtarget face image data is improved.

FIG. 11 is a schematic structural diagram of another image fusion deviceaccording to an embodiment of this application. As shown in FIG. 11, theimage fusion device 1 of this embodiment of this application mayinclude: a data obtaining module 11, a source grid generating module 12,a target grid generating module 13, a target data generating module 14,an effect adding module 15, a position adjustment module 16, and an edgefilling module 17.

The data obtaining module 11 is configured to obtain source face imagedata of a current to-be-fused image and resource configurationinformation of a current to-be-fused resource.

In specific implementation, the data obtaining module 11 may obtainsource face image data of a current to-be-fused image and resourceconfiguration information of a current to-be-fused resource. The sourceface image data may be face image data of a photo or a video taken by auser currently by using the image fusion device or selected from analbum of the image fusion device. The current to-be-fused resource maybe a resource model used for photo retouching that is in a photoretouching terminal application (for example, XXtupic, XXshop, orXXcamera) and that is selected by the user currently, for example, ananime character image or a celebrity photo. The resource configurationinformation may include 3D avatar information (for example, may be afile in obj format, and the file may include information representingresource face-related data, such as resource face image data, resourcecomplexion data, and a resource face three-dimensional grid) of thecurrent to-be-fused resource, information indicating a head effect in afinal result image (the information may include an orientation (Eulerangle pitch, yaw, and roll), a central position (a specified position ofthe final result image), scale information and camera informationmatching the scale information (for example, the information is depictedby using a perspective matrix), and the like of a resource 3D avatar ina world coordinate system), a 2D sticker and a 3D sticker, and a fusiondegree alpha of a user face and a resource face (all frames may have thesame fusion degree or each frame has a different fusion degree).

According to this embodiment of this application, the scale of theresource face image data for generating the target face image data andthe scale of the source face image data need to correspond to the samedimension, but facial concavity and plumpness of them may not becompletely consistent. Therefore, after the source face image data andthe resource face image data are obtained, the scale of the source faceimage data may be adjusted according to the scale of the resource faceimage data, to make the source face image data and the resource faceimage data correspond to the same dimension.

The source grid generating module 12 is configured to perform imagerecognition processing on the source face image data, to obtain sourceface feature points corresponding to the source face image data, andgenerate a source face three-dimensional grid of the source face imagedata according to the source face feature points.

According to this embodiment of this application, the source gridgenerating module 12 may perform image recognition processing on thesource face image data, to obtain source face feature pointscorresponding to the source face image data, and generate a source facethree-dimensional grid of the source face image data according to thesource face feature points.

FIG. 12 is a schematic structural diagram of a source grid generatingmodule according to an embodiment of this application. As shown in FIG.12, the source grid generating module 12 may include a feature pointobtaining unit 121 and a source grid generating unit 122.

The feature point obtaining unit 121 is configured to perform imagerecognition processing on the source face image data, to obtainreference feature points of the source face image data.

In specific implementation, the feature points obtaining unit 121 mayperform image recognition processing on the source face image data, toobtain reference feature points of the source face image data. The imagerecognition processing may be a process of performing recognition andfacial feature location on a user face in a photo by using a facedetection technology (for example, face detection provided by thecross-platform computer vision library OpenCV, the new vision serviceplatform Face++, or the Utu face detection). The reference featurepoints may be reference points indicating facial features, for example,points such as a facial contour, an eye contour, a nose, and lips, andmay be 83 reference points, or may be 68 reference points shown in FIG.3. The specific point quantity may be determined by developers accordingto requirements.

The source grid generating unit 122 is configured to performthree-dimensional depth information extraction on the reference featurepoints, to obtain source face feature points corresponding to thereference feature points, and generate a source face three-dimensionalgrid according to the source face feature points.

In specific implementation, the source grid generating unit 122 mayperform three-dimensional depth information extraction on the referencefeature points, to obtain source face feature points corresponding tothe reference feature points, and generate a source facethree-dimensional grid according to the source face feature points. Thethree-dimensional depth information extraction may be a process ofdeducing feature points that can reflect the source face image data in athree-dimensional model based on the foregoing reference feature pointsby matching facial feature points of a standard three-dimensional model.The source face feature points may be further deepened points based onthe reference feature points. For example, through the three-dimensionaldepth information extraction on the foregoing 68 or 83 reference points,1000 deepened source face feature points may be obtained, which may bevertexes of triangular patches in FIG. 4. The source facethree-dimensional grid may be a 3D face grid model corresponding to theface in the source face image data, and may be a 3D grid model of a userface formed by connecting the source face feature points, for example, a3D face grid shown in FIG. 4 or similar to a 3D face grid of only halfof a face in FIG. 4.

The target grid generating module 13 is configured to perform gridfusion by using the resource face three-dimensional grid and the sourceface three-dimensional grid to generate a target face three-dimensionalgrid.

According to this embodiment of this application, because scales of thesource face image data and the resource face image data are in the samedimension, dimensions of the source face three-dimensional grid and theresource face three-dimensional grid are also in the same dimension.

In specific implementation, the target grid generating module 13 mayperform grid fusion by using the resource face three-dimensional gridand the source face three-dimensional grid to generate a target facethree-dimensional grid. According to this embodiment of thisapplication, the resource face three-dimensional grid may be similar tothe foregoing source face three-dimensional grid, may be a 3D face gridmodel corresponding to the face in the resource face image data, and maybe a 3D face grid shown in FIG. 4 or a 3D face grid obtained after theorientation changes according to a Euler angle in a world coordinatesystem based on a 3D face grid shown in FIG. 4. The target facethree-dimensional grid may be a 3D face grid of the target face imagedata. According to this embodiment of this application, when theresource face three-dimensional grid is the 3D face grid shown in FIG.4, the resource face three-dimensional grid finally projected on a 2Dimage may present a front face effect, and when the resource facethree-dimensional grid is the 3D face grid obtained after theorientation changes according to a Euler angle in a world coordinatesystem based on the 3D face grid shown in FIG. 4, the resource facethree-dimensional grid finally projected on a 2D image may present aside face effect.

In this embodiment of this application, the target grid generatingmodule 13 may calculate target face feature points of a target faceaccording to the face feature points in the source facethree-dimensional grid and the resource face three-dimensional grid, andthen generate a target face three-dimensional grid according to thecalculated target face feature points. For example, a 3D source facethree-dimensional grid (namely, a 3D face grid of a user) has 1000source face feature points with depth information, which are marked tobe green, a 3D resource face three-dimensional grid has 1000 resourceface feature points with depth information, which are marked to be blue,average points of each corresponding point of the 1000 face featurepoints of the user and each corresponding point of the 1000 face featurepoints of the resource (corresponding points at the same position areaveraged, and there are a total of 1000 point pairs) are marked to bered, and the finally generated 1000 red face feature points are thetarget face feature points. The foregoing 1000 red face feature pointsmay form more than 1900 triangles, and a face three-dimensional griddepicted by corresponding more than 1900 triangular patches is thetarget face three-dimensional grid. According to this embodiment of thisapplication, the image fusion device 1 may use algorithms such as an MLSmethod, affine transformation, and image distortion, to make facialfeature positions of the source face image data and the resource faceimage data tend to facial feature positions indicated by the foregoingred face feature points, namely, the target face feature points, toachieve the objective of face fusion.

The target data generating module 14 is configured to perform facecomplexion fusion on the target face three-dimensional grid by using thesource complexion data of the source face image data and the resourcecomplexion data of the resource face image data, to generate fusedtarget face image data.

According to this embodiment of this application, the target datagenerating module 14 may perform face complexion fusion on the targetface three-dimensional grid by using source complexion data of thesource face image data and resource complexion data of the resource faceimage data, to generate fused target face image data.

FIG. 13 is a schematic structural diagram of a target data generatingmodule according to an embodiment of this application. As shown in FIG.13, the target data generating module 14 may include: a complexion dataobtaining unit 141, a candidate data generating unit 142, and a targetdata generating unit 143.

The complexion data obtaining unit 141 is configured to performcomplexion balance processing on the source face image data, to obtainaverage complexion data of the source face image data.

According to this embodiment of this application, if types of the sourceface three-dimensional grid and the resource face three-dimensional gridare inconsistent, grid supplementing needs to be performed on the sourceface three-dimensional grid, to generate a candidate facethree-dimensional grid. The types may be grid elements in the sourceface three-dimensional grid and the resource face three-dimensionalgrid. Candidate complexion data of the candidate face three-dimensionalgrid may be symmetrical source face image data, namely, the candidatecomplexion data of the candidate face three-dimensional grid may beregarded as the source complexion data. For example, if the finallygenerated candidate face image data is the face image in FIG. 5A, theface complexion in FIG. 5A is consistent with the face complexion inFIG. 5B.

In this embodiment of this application, the candidate complexion data ofthe candidate face three-dimensional grid may include two parts, namely,complexion data of a part of the candidate face three-dimensional gridmatching the source face three-dimensional grid is the source complexiondata, and complexion data of a part of the candidate facethree-dimensional grid not matching the source face three-dimensionalgrid is average complexion data. In specific implementation, thecomplexion data obtaining unit 141 may perform complexion balanceprocessing on the source face image data, to obtain average complexiondata of the source face image data. The complexion balance processingmay be a process of removing effects such as shadows caused by light orthe like from the source face image data to obtain a complexion averagevalue. The average complexion data may be a pixel point set formed byaverage values of pixel point data obtained after shadows of the sourcecomplexion data are removed.

The candidate data generating unit 142 is configured to performcomplexion filling on the candidate face three-dimensional grid based onthe source complexion data of the source face image data and the averagecomplexion data, to generate candidate face image data.

According to this embodiment of this application, the candidate datagenerating unit 142 may perform complexion filling on the candidate facethree-dimensional grid based on the source complexion data of the sourceface image data and the average complexion data, to generate candidateface image data. Complexion data of a part in the candidate facethree-dimensional grid matching the source face three-dimensional gridmay be filled by the source complexion data of the source face imagedata, and complexion data of a part in the candidate facethree-dimensional grid not matching the source face three-dimensionalgrid may be filled by the average complexion data. The candidatecomplexion data of the candidate face image data may include the sourcecomplexion data and the average complexion data. For example, if thefinally generated candidate face image data is the face image in FIG.5A, the complexion of the right face in FIG. 5A is the face complexionin FIG. 5B, and the complexion of the left face in FIG. 5A is complexionobtained after the face complexion in FIG. 5B is averaged.

The target data generating unit 143 is configured to perform facecomplexion fusion on the target face three-dimensional grid by using thecandidate complexion data of the candidate face image data and theresource complexion data of the resource face image data, to generatefused target face image data.

According to this embodiment of this application, after generating thetarget face three-dimensional grid, the image fusion device 1 needs toperform complexion filling on different triangular patches in the targetface three-dimensional grid to obtain the final target face image data.

According to this embodiment of this application, the target datagenerating unit 143 may perform face complexion fusion on the targetface three-dimensional grid by using the candidate complexion data ofthe candidate face image data and the resource complexion data of theresource face image data, to generate the fused target face image data.The candidate complexion data may be a set of candidate pixel pointsforming the candidate face image data, and the resource complexion datamay be a set of resource pixel points forming the resource face imagedata.

According to this embodiment of this application, the target datagenerating unit 143 may calculate target pixel points based on thecandidate pixel points and the resource pixel points and by using afusion degree, to generate target face image data according to thetarget pixel points. The fusion degree may be a fusion value setaccording to an empirical value, and the general value ranges from 0 to1.

According to this embodiment of this application, it may be set that:pixels of a feature point of a triangular patch in a candidate facethree-dimensional grid are: CandidateB, CandidateG, and CandidateR,pixels of a facial feature point at a corresponding position on thecorresponding triangular patch in a resource face three-dimensional gridare: ResourceB, ResourceG, and ResourceR, pixels of a feature point atthe corresponding position on the corresponding triangular patch in atarget face three-dimensional grid are: TargetB, TargetG, and TargetR,and the fusion degree in the resource configuration information is:alpha, there are:TargetB=(1.0−alpha)*CandidateB+alpha*ResourceBTargetG=(1.0−alpha)*CandidateG+alpha*ResourceGTargetR=(1.0−alpha)*CandidateR+alpha*ResourceR

Therefore, each pixel value of the target face image data may beobtained, to obtain the target face image data.

According to this embodiment of this application, by analyzing thefunction of the average complexion data in complexion fusion of thetarget face image data, authenticity of the finally obtained target faceimage data is increased.

The effect adding module 15 is configured to obtain a light source typecorresponding to the source face image data according to the sourcecomplexion data of the source face image data, and perform effect addingprocessing on the target face image data by using a lighting effectcorresponding to the light source type.

According to this embodiment of this application, after the target faceimage data is generated, the effect adding module 15 may obtain thelight source type corresponding to the source face image data accordingto the source complexion data of the source face image data. The effectadding module 15 may compare the average complexion with a complexion ofeach region in the source face image data, to obtain a region ofbrightness and the average complexion and a region of darkness and theaverage complexion, to further deduce the light source type, forexample, a plurality of point light sources or area light sources. Theeffect adding module 15 may also collect result images of the sourceface image data in different specified lighting situations through depthlearning, and then use the result images and the corresponding lightingsituations as training data of a DNN, to train a DNN model that canoutput a light source type and a light source position when a picture isgiven.

Further, the effect adding module 15 may perform effect addingprocessing on the target face image data by using the lighting effectcorresponding to the light source type. For example, if the light sourcetype corresponding to the source face image data is a point light sourcefrom the left face direction, the effect adding module 15 may add alighting effect of the point light source in the left face direction tothe finally obtained target face image data.

In this embodiment of this application, the effect adding module 15 mayfurther stick 2D and 3D stickers in the foregoing resource configurationinformation onto the target face image data according to thecorresponding position and zOrder, for example, wear a 3D glassessticker on the face of the generated target face image data.

The position adjustment module 16 is configured to adjust a currentdisplay position of the target face image data based on coordinateinformation indicated by the resource face image data.

According to this embodiment of this application, after the target faceimage data is generated, the position adjustment module 16 may furtheradjust the current display position of the target face image data basedon the coordinate information indicated by the resource face image data.For example, the obtained target face image data is placed on thespecified position according to the coordinate information (includingthe Euler direction and the central point) indicated by the resourceface image data in the foregoing resource configuration information andthe resource scale.

The edge filling module 17 is configured to perform, in a case that afirst display region of the target face image data is smaller than asecond display region of the source face image data, face edge fillingprocessing on a part in the second display region except the firstdisplay region.

According to this embodiment of this application, when the first displayregion of the finally generated target face image data is smaller thanthe second display region of the source face image data, the edgefilling module 17 may perform face edge filling processing on a part inthe second display region except the first display region. The firstdisplay region may be a range on a 2D screen to which the target faceimage data is mapped, the second display region may be a range on a 2Dscreen to which the source face image data is mapped, and that the firstdisplay region is smaller than the second display region may beexpressed as that the face of the target face image data is smaller thanthe face of the source face image data. The face edge filling processingmay be filling the part in the second display region except the firstdisplay region by using a filling algorithm (for example, the imagerestoration algorithm Inpainting provided by the OpenCV).

Further, after processing the additional technical effects on the targetface image data completely, the image fusion device 1 may output anddisplay the finally obtained target face image data.

According to this embodiment of this application, after the effectadding module 15, the position adjustment module 16, and the edgefilling module 17 perform the operations, the finally generated targetface image data may be three-dimensional face image data, or may betwo-dimensional face image data. When the target face image data istwo-dimensional face image data, because the process of generating thetarget image data is implemented based on the three-dimensional model,the effect of the finally formed target face image data is morerealistic in consideration of problems such as real light and shadows.

When the effect adding module 15, the position adjustment module 16, andthe edge filling module 17 perform the operations, one or more modulesmay be selected simultaneously to perform the operations.

In this embodiment of this application, by adding real lighting effectsto the generated target face image data, and adjusting a displayposition of face image data and filling a face edge region, real effectsof the finally outputted target face image data are further increased.

In a specific implementation of this embodiment of this application, thetarget data generating unit shown in FIG. 14 may include a pixel pointobtaining subunit 1431 and a target data generate subunit 1432.

The pixel point obtaining subunit 1431 is configured to obtain candidatepixel points in the candidate complexion data and resource pixel pointsin the resource complexion data.

According to this embodiment of this application, the candidatecomplexion data may be a set of candidate pixel points forming thecandidate face image data, and the resource complexion data may be a setof resource pixel points forming the resource face image data. The pixelpoint obtaining subunit 1431 may obtain candidate pixel points in thecandidate complexion data and resource pixel points in the resourcecomplexion data.

The target data generate subunit 1432 is configured to calculate targetpixel points based on the candidate pixel points and the resource pixelpoints and by using a fusion degree, to generate target face image dataaccording to the target pixel points.

In specific implementation, for the specific process of the target datagenerate subunit 1432 obtaining the target face image data based on thecandidate pixel points and the resource pixel points and by using thefusion degree, reference may be made to the description in the methodembodiment, and details are not provided herein again.

According to this embodiment of this application, by using a complexionfusion process accurate to pixel points, accuracy of the complexionfusion of the target face image data is improved.

In this embodiment of this application, source face image data of acurrent to-be-fused image and resource configuration information of acurrent to-be-fused resource are obtained, where the resourceconfiguration information includes resource face image data, resourcecomplexion data, and a resource face three-dimensional grid, then imagerecognition processing is performed on the source face image data, toobtain source face feature points corresponding to the source face imagedata, and a source face three-dimensional grid of the source face imagedata is generated according to the source face feature points, then gridfusion is performed by using the resource face three-dimensional gridand the source face three-dimensional grid to generate a target facethree-dimensional grid, and finally face complexion fusion is performedon the target face three-dimensional grid by using source complexiondata of the source face image data and resource complexion data of theresource face image data, to generate fused target face image data. Byanalyzing the process of fusing the resource face three-dimensional gridand the source face three-dimensional grid into the target facethree-dimensional grid based on a three-dimensional model, andperforming complexion fusion on the target face three-dimensional gridto generate target face image data, authenticity of the finally obtainedtarget face image data is improved. By analyzing the function of theaverage complexion data in complexion fusion of the target face imagedata, authenticity of the finally obtained target face image data isincreased. By adding real lighting effects to the generated target faceimage data, and adjusting a display position of face image data andfilling a face edge region, real effects of the finally outputted targetface image data are further increased. By using a complexion fusionprocess accurate to pixel points, accuracy of the complexion fusion ofthe target face image data is improved.

When face fusion is performed based on a 2D model, generally thereexists a situation in which a face angle of user image data and a faceangle of resource face image data do not match completely. For example,if the user image is half of the face, the resource image is the frontface or the user head turning left, the resource head turning right, andthe like. When the foregoing situation exists, the face fusion algorithmin the related art can obtain less user face information, and duringface fusion the less user face information will affect the finalmatching result, causing poor authenticity of the generated targetresult image.

To resolve the foregoing problem, an embodiment of this applicationprovides a schematic structural diagram of another image fusion device.For details, reference may be made to the schematic structural diagramshown in FIG. 10. The image fusion device may include: a data obtainingmodule 11, a source grid generating module 12, a target grid generatingmodule 13, and a target data generating module 14.

The data obtaining module 11 is configured to obtain source face imagedata of a current to-be-fused image and resource configurationinformation of a current to-be-fused resource.

In specific implementation, for the process of the data obtaining module11 obtaining the source face image data and the resource configurationinformation, reference may be made to the specific description in theforegoing method embodiment, and details are not provided herein again.

The source grid generating module 12 is configured to perform imagerecognition processing on the source face image data, to obtain sourceface feature points corresponding to the source face image data, andgenerate a source face three-dimensional grid of the source face imagedata according to the source face feature points.

In specific implementation, for the process of the source gridgenerating module 12 generating the source face three-dimensional grid,reference may be made to the specific description in the foregoingmethod embodiment, and details are not provided herein again.

The target grid generating module 13 is configured to perform gridsupplementing on the source face three-dimensional grid according to thesymmetry of the source face image data when it is detected that types ofthe source face three-dimensional grid and the resource facethree-dimensional grid are inconsistent, generate a candidate facethree-dimensional grid whose type is consistent with that of theresource face three-dimensional grid, and perform grid fusion by usingthe candidate face three-dimensional grid and the resource facethree-dimensional grid to generate a target face three-dimensional grid.

In specific implementation, when it is detected that types of the sourceface three-dimensional grid and the resource face three-dimensional gridare inconsistent, the target grid generating module 13 may perform gridsupplementing on the source face three-dimensional grid according to thesymmetry of the source face image data, generate candidate facethree-dimensional grid whose type is consistent with that of theresource face three-dimensional grid, and perform grid fusion by usingthe candidate face three-dimensional grid and the resource facethree-dimensional grid to generate a target face three-dimensional grid.

According to this embodiment of this application, because scales of thesource face image data and the resource face image data are in the samedimension, dimensions of the source face three-dimensional grid and theresource face three-dimensional grid are also in the same dimension.

According to this embodiment of this application, the resource facethree-dimensional grid may be similar to the foregoing source facethree-dimensional grid, may be a 3D face grid model corresponding to theface in the resource face image data, and may be a 3D face grid shown inFIG. 4 or a 3D face grid obtained after the orientation changesaccording to a Euler angle in a world coordinate system based on a 3Dface grid shown in FIG. 4. The target face three-dimensional grid may bea 3D face grid of the target face image data. When the resource facethree-dimensional grid is the 3D face grid shown in FIG. 4, the resourceface three-dimensional grid finally projected on a 2D image may presenta front face effect, and when the resource face three-dimensional gridis the 3D face grid obtained after the orientation changes according toa Euler angle in a world coordinate system based on the 3D face gridshown in FIG. 4, the resource face three-dimensional grid finallyprojected on a 2D image may present a side face effect.

According to this embodiment of this application, only when the types ofthe source face three-dimensional grid and the resource facethree-dimensional grid are consistent, a target face three-dimensionalgrid with good authenticity that is the same as a facial display regionof the source face three-dimensional grid and the resource facethree-dimensional grid can be generated according to a fusion algorithm.That the types are consistent may be that grid elements in the sourceface three-dimensional grid and the resource face three-dimensional gridare consistent, and the grid elements may be grid orientations or griddisplay regions of the source face three-dimensional grid and theresource face three-dimensional grid. For example, when the facialdisplay region indicated by the source face three-dimensional grid andthe facial display region indicated by the resource facethree-dimensional grid are consistent, and the source facethree-dimensional grid and the resource face three-dimensional grid aresimilar to the standard front face shown in FIG. 5A or similar to theside face shown in FIG. 5B, it may be regarded that the types of thesource face three-dimensional grid and the resource facethree-dimensional grid are consistent. When the types of the source facethree-dimensional grid and the resource face three-dimensional grid areinconsistent (for example, the source face three-dimensional grid issimilar to the side face shown in FIG. 5B, and the resource facethree-dimensional grid is similar to the standard front face shown inFIG. 5A), the image fusion device 1 may first perform grid supplementingon the source face three-dimensional grid.

In specific implementation, when the types of the source facethree-dimensional grid and the resource face three-dimensional grid areinconsistent, the target grid generating module 13 may perform gridsupplementing on the source face three-dimensional grid according to thesymmetry of the source face image data, to generate a candidate facethree-dimensional grid whose type is consistent with that of theresource face three-dimensional grid. In a normal situation, a face of aperson is symmetrical (nuances are ignored), and when the source faceimage data is side face image data (for example, the image in FIG. 5B),the source face three-dimensional grid generated after the image fusiondevice extracts the source face feature points is also for a side face.In this case, the target grid generating module 13 may supplement thesource face image data as a standard front face (for example, the imagein FIG. 5A is obtained after the image in FIG. 5B is supplemented)according to the face symmetry principle, and then generate a candidateface three-dimensional grid whose type is consistent with that of theresource face three-dimensional grid (the face three-dimensional gridcorresponding to FIG. 5A is the candidate face three-dimensional grid)according to the supplemented source face image data.

According to this embodiment of this application, left and right facialexpressions of image data of the standard front face obtained after thetarget grid generating module 13 supplements the source face image data(side face image data) according to the face symmetry principle areconsistent, and left and right facial expressions of the furtherobtained candidate face three-dimensional grid are also consistent.

In this embodiment of this application, when the resource facethree-dimensional grid is a standard front face whose left and rightfacial expressions are inconsistent, and the source face image data isside face image data, left and right facial expressions of the candidateface three-dimensional grid obtained after grid supplementing isperformed on the source face three-dimensional grid according to thesymmetry of the source face image data are inconsistent with left andright facial expressions of the resource face three-dimensional grid(for example, the left eye of the resource face indicated by theresource face image data is opened, and the right eye is closed, but theleft eye of the user face indicated by the source face image dataincluding only the left side face is opened. In this case, the right eyeof the source face three-dimensional grid supplemented according to thesymmetry of the source face image data is also opened, which isinconsistent with the right eye of the resource face three-dimensionalgrid). In this case, the target grid generating module 13 cannot performgrid fusion by using the candidate face three-dimensional grid and theforegoing resource face three-dimensional grid to generate the targetface three-dimensional grid. For the foregoing situation, the targetgrid generating module 13 may adjust the left and right facialexpressions of the candidate face three-dimensional grid to beconsistent with the expression of the resource face three-dimensionalgrid by using an expression migration algorithm, so that facialexpressions of the finally obtained candidate face three-dimensionalgrid and the foregoing resource face three-dimensional grid areconsistent.

Further, the target grid generating module 13 may perform grid fusion byusing the candidate face three-dimensional grid and the resource facethree-dimensional grid, to generate the target face three-dimensionalgrid. The types of the candidate face three-dimensional grid and theresource face three-dimensional grid are consistent, namely, theresource face three-dimensional grid and the candidate facethree-dimensional grid on the same facial position have correspondingfeature points. For example, the candidate face three-dimensional gridincludes feature points of corners of two eyes, and the resource facethree-dimensional grid also includes feature points of corners of twoeyes. The resource face three-dimensional grid and the candidate facethree-dimensional grid on the same facial position have correspondingfeature points. The target face three-dimensional grid may be a 3D facegrid of the target face image data.

In this embodiment of this application, the target grid generatingmodule 13 may calculate target face feature points of a target faceaccording to the face feature points in the candidate facethree-dimensional grid and the resource face three-dimensional grid, andthen generate the target face three-dimensional grid according to thecalculated target face feature points. For example, a 3D candidate facethree-dimensional grid (namely, a 3D face grid of a user) has 1000source face feature points with depth information, which are marked tobe green, a 3D resource face three-dimensional grid has 1000 resourceface feature points with depth information, which are marked to be blue,average points of each corresponding point of the 1000 face featurepoints of the user and each corresponding point of the 1000 face featurepoints of the resource (corresponding points at the same position areaveraged, and there are a total of 1000 point pairs) are marked to bered, and the finally generated 1000 red face feature points are thetarget face feature points. The foregoing 1000 red face feature pointsmay form more than 1900 triangles, and a face three-dimensional griddepicted by corresponding more than 1900 triangular patches is thetarget face three-dimensional grid. According to this embodiment of thisapplication, the image fusion device 1 may use algorithms such as an MLSmethod, affine transformation, and image distortion, to make facialfeature positions of the source face image data and the resource faceimage data tend to facial feature positions indicated by the foregoingred face feature points, namely, the target face feature points, toachieve the objective of face fusion.

The target data generating module 14 is configured to perform facecomplexion fusion on the target face three-dimensional grid by usingsource complexion data of the source face image data and resourcecomplexion data of the resource face image data, to generate fusedtarget face image data.

In specific implementation, for the process of the target datagenerating module 14 generating the target face image data, referencemay be made to the specific description in the foregoing methodembodiment, and details are not provided herein again.

An embodiment of this application further provides a computer storagemedium. The computer storage medium may store a plurality ofinstructions, the instructions being suitable for being loaded by aprocessor and performing the method steps in the foregoing embodimentsshown in FIG. 2 to FIG. 9. For the specific execution process, referencemay be made to the specific description of the embodiments shown in FIG.2 to FIG. 9, and details are not provided herein again.

FIG. 15 is a schematic structural diagram of a terminal according to anembodiment of this application. As shown in FIG. 15, the terminal 1000may include at least one processor 1001 such as a CPU, at least onenetwork interface 1004, a user interface 1003, a memory 1005, and atleast one communications bus 1002. The communications bus 1002 isconfigured to implement connection and communication between thecomponents. The user interface 1003 may include a display, a keyboard,and optionally, the user interface 1003 may further include a standardwired interface and a standard wireless interface. The network interface1004 may include a standard wired interface and a wireless interface(such as a WI-FI interface). The memory 1005 may be a high-speed RAMmemory, or may be a non-volatile memory, for example, at least onemagnetic disk memory. The memory 1005 may further be at least onestorage apparatus located far away from the processor 1001. As shown inFIG. 15, the memory 1005, which is used as a computer storage medium,may include an operating system, a network communications module, a userinterface module, and a face fusion application.

In the terminal 1000 shown in FIG. 15, the user interface 1003 is mainlyconfigured to: provide an input interface for a user, and obtain dataentered by the user. The network interface 1004 is mainly configured tocommunicate data with a user terminal. The processor 1001 may beconfigured to invoke the face fusion application stored in the memory1005 and specifically perform the following operations:

obtaining source face image data of a current to-be-fused image andresource configuration information of a current to-be-fused resource,the resource configuration information including resource face imagedata, resource complexion data, and a resource face three-dimensionalgrid;

performing image recognition processing on the source face image data,to obtain source face feature points corresponding to the source faceimage data, and generating a source face three-dimensional grid of thesource face image data according to the source face feature points;

performing grid fusion by using the resource face three-dimensional gridand the source face three-dimensional grid to generate a target facethree-dimensional grid; and

performing face complexion fusion on the target face three-dimensionalgrid by using source complexion data of the source face image data andresource complexion data of the resource face image data, to generatefused target face image data.

In an embodiment, when performing image recognition processing on thesource face image data, to obtain source face feature pointscorresponding to the source face image data, and generating a sourceface three-dimensional grid of the source face image data according tothe source face feature points, the processor 1001 specifically performsthe following operations:

performing image recognition processing on the source face image data,to obtain reference feature points of the source face image data; and

performing three-dimensional depth information extraction on thereference feature points, to obtain source face feature pointscorresponding to the reference feature points, and generating a sourceface three-dimensional grid according to the source face feature points.

In an embodiment, when performing grid fusion by using the resource facethree-dimensional grid and the source face three-dimensional grid togenerate a target face three-dimensional grid, the processor 1001specifically performs the following operation:

performing grid supplementing on the source face three-dimensional gridaccording to the symmetry of the source face image data when it isdetected that types of the source face three-dimensional grid and theresource face three-dimensional grid are inconsistent, generating acandidate face three-dimensional grid whose type is consistent with thatof the resource face three-dimensional grid, and performing grid fusionby using the candidate face three-dimensional grid and the resource facethree-dimensional grid to generate a target face three-dimensional grid.

In an embodiment, when performing face complexion fusion on the targetface three-dimensional grid by using source complexion data of thesource image data and resource complexion data of the resource faceimage data, to generate fused target face image data, the processor 1001specifically performs the following operations:

performing complexion balance processing on the source face image data,to obtain average complexion data of the source face image data;

performing complexion filling on the candidate face three-dimensionalgrid based on the source complexion data of the source face image dataand the average complexion data, to generate candidate face image data;and

performing face complexion fusion on the target face three-dimensionalgrid by using candidate complexion data of the candidate face image dataand the resource complexion data of the resource face image data, togenerate the fused target face image data, where

the candidate complexion data includes the source complexion data andthe average complexion data.

In an embodiment, when performing face complexion fusion on the targetface three-dimensional grid by using candidate complexion data of thecandidate face image data and the resource complexion data of theresource face image data, to generate the fused target face image data,the processor 1001 specifically performs the following operations:

obtaining candidate pixel points in the candidate complexion data andresource pixel points in the resource complexion data; and

calculating target pixel points based on the candidate pixel points andthe resource pixel points and by using a fusion degree, and generatingthe target face image data according to the target pixel points.

In an embodiment, the processor 1001 is further configured to performthe following operation:

obtaining a light source type corresponding to the source face imagedata according to the source complexion data of the source face imagedata, and performing effect adding processing on the target face imagedata by using a lighting effect corresponding to the light source type.

In an embodiment, the processor 1001 is further configured to performthe following operation:

adjusting a current display position of the target face image data basedon coordinate information indicated by the resource face image data.

In an embodiment, the processor 1001 is further configured to performthe following operation:

performing, in a case that a first display region of the target faceimage data is smaller than a second display region of the source faceimage data, face edge filling processing on a part in the second displayregion except the first display region.

In this embodiment of this application, source face image data of acurrent to-be-fused image and resource configuration information of acurrent to-be-fused resource are obtained, where the resourceconfiguration information includes resource face image data, resourcecomplexion data, and a resource face three-dimensional grid, then imagerecognition processing is performed on the source face image data, toobtain source face feature points corresponding to the source face imagedata, and a source face three-dimensional grid of the source face imagedata is generated according to the source face feature points, then gridfusion is performed by using the resource face three-dimensional gridand the source face three-dimensional grid to generate a target facethree-dimensional grid, and finally face complexion fusion is performedon the target face three-dimensional grid by using source complexiondata of the source face image data and resource complexion data of theresource face image data, to generate fused target face image data. Byanalyzing the process of fusing the resource face three-dimensional gridand the source face three-dimensional grid into the target facethree-dimensional grid based on a three-dimensional model, andperforming complexion fusion on the target face three-dimensional gridto generate target face image data, authenticity of the finally obtainedtarget face image data is improved. By analyzing the function of theaverage complexion data in complexion fusion of the target face imagedata, authenticity of the finally obtained target face image data isincreased. By adding real lighting effects to the generated target faceimage data, and adjusting a display position of face image data andfilling a face edge region, real effects of the finally outputted targetface image data are further increased. By using a complexion fusionprocess accurate to pixel points, accuracy of the complexion fusion ofthe target face image data is improved.

A person of ordinary skill in the art may understand that all or some ofthe processes of the methods in the embodiments may be implemented by acomputer program instructing relevant hardware. The program may bestored in a computer readable storage medium. When the program isexecuted, the procedures of the methods in the embodiments areperformed. The storage medium may be a magnetic disk, an optical disc, aread-only memory (ROM), or a random access memory (RAM).

The foregoing disclosure is merely embodiments of this application, andcertainly is not intended to limit the protection scope of thisapplication. Therefore, equivalent variations made in accordance withthe claims of this application shall fall within the scope of thisapplication.

What is claimed is:
 1. An image fusion method performed at a computing device having a processor and memory and a plurality of programs stored in the memory, the method comprising: obtaining source face image data of a current to-be-fused image and resource configuration information of a current to-be-fused resource, the resource configuration information comprising resource face image data, resource complexion data, and a resource face three-dimensional grid; obtaining source face feature points from the source face image data through image recognition; generating a source face three-dimensional grid of the source face image data according to the source face feature points; performing grid fusion to the resource face three-dimensional grid and the source face three-dimensional grid to generate a target face three-dimensional grid; and performing face complexion fusion on the target face three-dimensional grid by using source complexion data of the source face image data and the resource complexion data of the resource face image data, to generate fused target face image data.
 2. The method according to claim 1, wherein the obtaining source face feature points from the source face image data through image recognition comprises: obtaining reference feature points of the source face image data through the image recognition; extracting three-dimensional depth information from the reference feature points, to obtain the source face feature points corresponding to the reference feature points; and generating the source face three-dimensional grid according to the source face feature points.
 3. The method according to claim 1, wherein the performing grid fusion to the resource face three-dimensional grid and the source face three-dimensional grid to generate a target face three-dimensional grid comprises: performing grid supplementing on the source face three-dimensional grid according to a symmetry of the source face image data after detecting that types of the source face three-dimensional grid and the resource face three-dimensional grid are inconsistent; generating a candidate face three-dimensional grid whose type is consistent with that of the resource face three-dimensional grid; and performing the grid fusion to the candidate face three-dimensional grid and the resource face three-dimensional grid to generate the target face three-dimensional grid.
 4. The method according to claim 3, wherein the performing face complexion fusion on the target face three-dimensional grid by using source complexion data of the source face image data and resource complexion data of the resource image data, to generate fused target face image data comprises: performing complexion balance on the source face image data, to obtain average complexion data of the source face image data; performing complexion filling on the candidate face three-dimensional grid based on the source complexion data of the source face image data and the average complexion data, to generate candidate face image data; and performing face complexion fusion on the target face three-dimensional grid using candidate complexion data of the candidate face image data and the resource complexion data of the resource face image data, to generate the fused target face image data, wherein the candidate complexion data comprises the source complexion data and the average complexion data.
 5. The method according to claim 4, wherein the performing face complexion fusion on the target face three-dimensional grid using candidate complexion data of the candidate face image data and the resource complexion data of the resource face image data, to generate the fused target face image data comprises: obtaining candidate pixel points in the candidate complexion data and resource pixel points in the resource complexion data; calculating target pixel points based on the candidate pixel points and the resource pixel points using a fusion degree; and generating the target face image data according to the target pixel points.
 6. The method according to claim 1, further comprising: obtaining a light source type corresponding to the source face image data according to the source complexion data of the source face image data; and adding a light effect on the target face image data corresponding to the light source type.
 7. The method according to claim 1, further comprising: adjusting a current display position of the target face image data based on coordinate information indicated by the resource face image data.
 8. The method according to claim 1, further comprising: in accordance with a determination that a first display region of the target face image data is smaller than a second display region of the source face image data, filling face edge on a part of the second display region except the first display region.
 9. A computing device, comprising: a processor and memory, the memory storing a plurality of computer programs, wherein the computer programs, when executed by the processor, perform operations including: obtaining source face image data of a current to-be-fused image and resource configuration information of a current to-be-fused resource, the resource configuration information comprising resource face image data, resource complexion data, and a resource face three-dimensional grid; obtaining source face feature points from the source face image data through image recognition; generating a source face three-dimensional grid of the source face image data according to the source face feature points; performing grid fusion to the resource face three-dimensional grid and the source face three-dimensional grid to generate a target face three-dimensional grid; and performing face complexion fusion on the target face three-dimensional grid by using source complexion data of the source face image data and the resource complexion data of the resource face image data, to generate fused target face image data.
 10. The computing device according to claim 9, wherein the obtaining source face feature points from the source face image data through image recognition comprises: obtaining reference feature points of the source face image data through the image recognition; extracting three-dimensional depth information from the reference feature points, to obtain the source face feature points corresponding to the reference feature points; and generating the source face three-dimensional grid according to the source face feature points.
 11. The computing device according to claim 9, wherein the performing grid fusion to the resource face three-dimensional grid and the source face three-dimensional grid to generate a target face three-dimensional grid comprises: performing grid supplementing on the source face three-dimensional grid according to a symmetry of the source face image data after detecting that types of the source face three-dimensional grid and the resource face three-dimensional grid are inconsistent; generating a candidate face three-dimensional grid whose type is consistent with that of the resource face three-dimensional grid; and performing the grid fusion to the candidate face three-dimensional grid and the resource face three-dimensional grid to generate the target face three-dimensional grid.
 12. The computing device according to claim 11, wherein the performing face complexion fusion on the target face three-dimensional grid by using source complexion data of the source face image data and resource complexion data of the resource image data, to generate fused target face image data comprises: performing complexion balance on the source face image data, to obtain average complexion data of the source face image data; performing complexion filling on the candidate face three-dimensional grid based on the source complexion data of the source face image data and the average complexion data, to generate candidate face image data; and performing face complexion fusion on the target face three-dimensional grid using candidate complexion data of the candidate face image data and the resource complexion data of the resource face image data, to generate the fused target face image data, wherein the candidate complexion data comprises the source complexion data and the average complexion data.
 13. The computing device according to claim 12, wherein the performing face complexion fusion on the target face three-dimensional grid using candidate complexion data of the candidate face image data and the resource complexion data of the resource face image data, to generate the fused target face image data comprises: obtaining candidate pixel points in the candidate complexion data and resource pixel points in the resource complexion data; calculating target pixel points based on the candidate pixel points and the resource pixel points using a fusion degree; and generating the target face image data according to the target pixel points.
 14. The computing device according to claim 9, wherein the operations further comprise: obtaining a light source type corresponding to the source face image data according to the source complexion data of the source face image data; and adding a light effect on the target face image data corresponding to the light source type.
 15. The computing device according to claim 9, wherein the operations further comprise: adjusting a current display position of the target face image data based on coordinate information indicated by the resource face image data.
 16. The computing device according to claim 9, wherein the operations further comprise: in accordance with a determination that a first display region of the target face image data is smaller than a second display region of the source face image data, filling face edge on a part of the second display region except the first display region.
 17. A non-transitory computer storage medium, storing a plurality of instructions, the instructions being configured for, when executed by a processor a computing device, perform operations including: obtaining source face image data of a current to-be-fused image and resource configuration information of a current to-be-fused resource, the resource configuration information comprising resource face image data, resource complexion data, and a resource face three-dimensional grid; obtaining source face feature points from the source face image data through image recognition; generating a source face three-dimensional grid of the source face image data according to the source face feature points; performing grid fusion to the resource face three-dimensional grid and the source face three-dimensional grid to generate a target face three-dimensional grid; and performing face complexion fusion on the target face three-dimensional grid by using source complexion data of the source face image data and the resource complexion data of the resource face image data, to generate fused target face image data.
 18. The non-transitory computer storage medium according to claim 17, wherein the operations further comprise: obtaining a light source type corresponding to the source face image data according to the source complexion data of the source face image data; and adding a light effect on the target face image data corresponding to the light source type.
 19. The non-transitory computer storage medium according to claim 17, wherein the operations further comprise: adjusting a current display position of the target face image data based on coordinate information indicated by the resource face image data.
 20. The non-transitory computer storage medium according to claim 17, wherein the operations further comprise: in accordance with a determination that a first display region of the target face image data is smaller than a second display region of the source face image data, filling face edge on a part of the second display region except the first display region. 